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ABSTRACT 
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the Delphi Method, an interactive panel of experts was convened to 
work toward a consensus on the role of critical thinking (CT) in 
educational assessment and instruction. In Delphi research, experts 
participate in several rounds of questions that require thoughtful 
and detailed responses. Panelists work toward consensus by sharing 
reasoned Opinions and reconsidering the opinions with regard to 
comments, objections, and arguments offered by other experts. A total 
of 46 scholars, educators, and leading figures in CT theory and CT 
assessment research were gathered for the panel meetings. About half 
of the ranelists were primarily affiliated with philosophy 
Gepartments; the others were affiliated with education, social 
sciences, or physical sciences. Recommendations resulting from the 
discussion rounds address the cognitive skill dimension of CT, the 
dispositional dimension of CT, and specific recommendations on CT 
instruction and assessment, including development of a CT curriculum. 
A discussion of commercially available CT assessment tools, a 
bibliography with an emphasis on assessment, and a set of letters 
which chronicle the progress of the Delphi research group are 
appended. (TJH) 
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CRITICAL THINKING: A STATEMENT OF EX?PFRT CONSENSUS 
FOR PURPOSES OF EDUCATIONAL ASSESSMENT AND INSTRUCTION 


zz The Critical Thinking Movement and CT Assessment 


From New Jersey to California, and from Newfoundland to Florida, 
leaders in the critical thinking movement have advocated major 
educational reform. They have argued that effective and meaningful 
education requires that curricular, pedagogical and assessment strategies 
at all levels of education be coordinated so as to foster in students 
those cognitive skills and habits of inquiry associated with critical 
thinking. They have made the case that educating students to be critical 
thinkers is vital for the students themselves and for society in general, 
(Ennis, 1962, 1981, 1986; Passmore, 1967; Schievella, 1958; Sheffler, 

1973; Lipman, 1977; Siegel, 1980, 1988; Gardner, 1983; Arons, 1983; 
Beyer, 1985; Costa, 1985; Quellmalz, 1983, 1985; Scriven, 1985; 
Sternberg, 1985; Ruggiero, 1938; Paul, 1988 (a) and (b); etc.). 

The arguments for critical thinking have been successful. 

After decades of relative neglect, the eighties witnessed a growing 
accord that the heart of education lies exactly where traditional 
advocates of a liberal education always said it was -- in the processes 
of inquiry, learning and thinking rather than in the accumulation of 


disjointed skills and senescent information. The critical thinking 


movement gained momentum throughout the decade. Conferences and position 


papers led to the development of college level critical thinking (CT) 
courses. In elementary and secondary schools (K-12) teachers revised 
lesson Plans to incorporate CT objectives. In the span of a few years 
publishing CT textbooks and offering CT staff development programs became 


growth industries. The CT movement enjoyed major success when 
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universities introduced CT requirements into their general education 
programs and state departments of education targeted CT in their 
curricular frameworks and their standardized testing programs. By the 
decade’s end CT could no longer be characterized as a cottage industry. 

With success come questions: Not new ones necessarily, but, because 
of the expectations which have been raised and the investments being 
proposed, vexing ones. Intuitively, CT instruction should focus on how 
students approach a question and reason about it. CT pedagogy should 
develop in students those cognitive skills and affective dispositions 
which characterize the good critical thinker. Rather than or in addition 
to targeting whether a given answer is correct, CT assessment should 
target the quality of the critical thinking the students put into 
arriving at that answer. Thus, for all of their successes, CT experts 
find they must continue to address some fundamental academic concerns. 
What exactly are those skills and dispositions which characterize CT? 
What are some effective ways to teach CT? And haw can CT, particularly 
if it becomes a campus-wide, district-wide or statewide requirement, ve 
assessed? 

When these academic questions are asked by the individual professor 
ur teacher seeking to introduce CT into her own classroom, they are 
difficull enough. But the questions take on social, fiscal, and 
political dimensions when asked by campus curriculum committees, school 
district offices, boards of education, and the educational testing and 
publishing industries. This is not to say that the experts find these 
questions insurmountable. On the contrary, CT experts have worked with 
their colleagues in the education community on some remarkable projects. 
For example, California and New Jersey have established ways of 


introducing CT into their curricular frameworks and statewide testing 


C; 





programs. The twenty-campus California State University system, which 
enrolls hundreds of thousands of students, has established a process for 
the approval of CT courses for its general education requirement. 

Given the central role played by philosophers in articulating the 
value, both individual and social, of CT, in analyzing the concept of CT, 
in designing college level academic programs in CT, and in assisting with 
efforts to introduce CT into the K-12 curriculum, it is little wonder 
that the American Fhilosophical Association, through its Committee on 
Fre-College Philosophy, has taken an interest in the CT movement and its 
impact on the profession. In December of 1987 that committee asked this 


investigator to make a systematic inquiry into the current state of CT 


and CT assessment. 





TABLE 1 


















CONSENSUS STATEMENT REGARDING CRITICAL 
THINKING AND THE IDEAL CRITICAL THINKER 


We understand critical thinking to be purposeful, self-regulatory judgment 
which results in interpretation, analysis, evaluation, and inference, as well as 
explanation of the evidential, conceptual, methodological, criteriological, or 
contextual considerations upon which that judgment is based. CT is essential as a tool 
of inquiry. Assuch, CT is a liberating force in education and a powerful resource in one’s 
personal and civic life. While not synonymous with good thinking, CT is a pervasive 
and self-rectifying human phenomenon. The ideal critical thinker is habitually 
inquisitive, well-informed, trustful of reason, open-minded, flexible, fair-minded in 
evaluation, honest in facing personal biases, prudent in making judgments, willing to 
reconsider, clear about issues, orderly in complex matters, diligent in seeking relevant 
information, reasonable in the selection of criteria, focused in inquiry, and persistent 
in seeking results which are as precise as the subject and the circumstances of inquiry 
permit. Thus, educating good critical thinkers means working toward this ideal. It 
combines developing CT skills with nurturing those dispositions which consistently 
yield useful insights and which are the basis of a rational and democratic society. 





As Table 1 suggests, a key result of inquiry is the articulation by 
a panel of CT experts of a conceptualization of CT it terms of two 


dimensions: cognitive skills and affective dispositions. Section II of 
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this report describes the Delphi research methodology. Section III 
address the skill dimension of CT, and Section IV the dispositional 
dimension of CT. Research findings are presented throughout the report, 
both in the text and in tabular form. Six recommendations are presented 
in Sections III and IV so they can be related most sensibly with their 
rationale. Nine additional recommendations which pertain specifically to 


CT instruction and assessment are presented in Section V. 


Il_-- Research Methodology and Purpose 


The Committee on Pre-College Philosophy suggested several persons 
with special expertise in CT and CT whom this investigator might contact 
as part of the inquiry into the controversial issues known to lie at the 
heart of the profession’s concern. This investigator decided to employ 
the powerful qualitative research methodology known as the Delphi Method. 
The Delphi Method requires the formation of an interactive panel of 
experts. These persons must be willing to share their expertise and work 
toward a consensus resolution of matters of opinion. Using the first 


group of experts to nominate others, the Delphi panel soon took shape. 


. In all forty-six persons, widely recognized by their professional 


colleagues to have special experience and expertise in CT anstruction, 
assessment or theory, made the commitment to farticipate in this Delphi 
project. If it were not for their canscienticus effort, (for which this 


investigator is extremely appreciative), the cansensus expressed in this 


‘report could not have teen reached. 


In Delphi research experts participate in several rounds of 
questions which call for thoughtful and detailed responses. Achieving a 


monsensus Of expert opinion using the Delphi Method is not a matter of 
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voting or tabulating quantitative data. Rather the expert panelists work 
toward consensus by sharing their reasoned opinions and being willing to 
reconsider them in the light of the comments, objections and arguments 
offered by other experts. In Delphi research, once an expert cxpresses 
an opinion, even a dissenting one, it becomes a factor ii: the mix and 
flow of all subsequent argument and thought. To circumvent undue 
influence arising from any given expert's m ofessional status, each round 
of questions is initiated by the project director and all responses are 
coordinated through that person. The project director circulates to the 
entire panel direct Ea and synthesized responses, with the names 
of their autho: s ‘removed. 

The eerk panelists themselves, through the thoughtfulness and 
porsuasiveness of their written responses, shape the line of inquiry. 
The project director endeavors to frame questions which respond to the 
direction panel debate is taking and lead the conversation toward 
fruitful resolution. As the inquiry proceeds, the project director 
assists the panelists with bibliographies and alerts them to other useful 
sources of relevant information. As areas of accord or disagreement 
anerge these are presented to the panel in the form of drafts of 
preliminary findings or crucial follow-up questions. The process 
terminates when the project director cetermines that sufficient accord 
has been reached for areas of consens, . to be made public. Delphi 
findings also include descriptions of residual disagreement and 
statements of minority opinion. 

A clear and accurate conceptualization of CT is absolutely essential 
for the development of valid CT assessment tools and effective CT 
instructional programs. ‘ith this in mind, and. recognizing that 


divergent conceptualizations of CT have hindered curricular and 
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assessment efforts, early in the Delphi process the Panel decided its 
mast worthwhile contribution could be the articulation of a clear and 
correct conceptualization cf CT. The expert panelists devoted their 
major effort toward that end. The experts hoped that by coming to 
consensus they could offer educators interested in CT assessment or 
instruction a pontaptuatteabien of CT of sufficient clarity, accuracy and 
richness to warrant their serious attention. 

To balance the theoretical with the practical, the experts asked 
themselves what a generally educated college lower division level 
critical thinker should be able to do. However, they did not attempt to 
describe the typical college level critical thinker. It soon became 
evident that the experts were actually articulating an ideal. It may be 
that no person is fully adept at all the skills and sub-skills the 
experts found to be central to CT. It may be that no person has fully 
cultivated all the affective dispositions which characterize a good 
critical thinker. Alsa humans compartmentalize their lives in ways that 
CT is more active and evident in some areas than in others. This gives 
no more reason to abandon the effort to infuse CT into the educational 
system than that knowing no friendship is perfect gives one reason to 
despair of having friends. The experts’ purpose in putting the ideal 
before the education community is that it should serve as a rich and 
worthy goal guiding CT assessment and curriculum development at all 


educational levels. 
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TABLE 2 
PROJECT HISTORY 


Round 1 (Feb. 11, 1988) and Round 2 (Mar. 14, 1988) initiated the Delphi 
process. In both rounds panelists were invited to nominate other CT experts to join 
in this research project. The experts reached consensus on the working assumption that 
“the concept of CT could be made operational to the extent that important parts of CT 
could be assessed velidly and reliably.” The experts agreed to begin their analysis 
of CT by “identifying the core elements of CT whivh might reasonably be expected 
at the freshman and sophomore general education college level.” The rationale for this 
decision was that the college levei theoretical construct of CT could reasonably be used 
to guide what might be said about CT at the K-12 level. Also the panelists noted that 
most of the participating experts had greater experience at the college level than in K- 
12 education. 


Round 3 (May 4, 1988) was an open-ended invitation for experts to write their own 


list of the operations which they conceived of as central to CT. The first synthesis ofthis | 


input was presented for expert review in Round 4 (Sept. 23, 1988). This synthesis 
focused on the skill dimension of CT. Round 4 invited responses regarding each skill 
and sub-skill identified, a proposed [and ultimately rejected] input/output model of CT 
operations, a list of closely related cognitive operations which might or might not be 
distinguished from CT, a general statement regarding what a skill is and how one is 
taught, and a list of caveats and cautions regarding CT instruction and assessment. 


Round SA (Feb. 28, 1989) reviewed the definitions and classification of CT 
cognitive skills in the light of expert responses to Round4. Rovad 5B (also Feb. 28, 1989) 
proposed statements regarding the dispositional dimension of CT and about its possible 
normative connotations. Round SC (Mar. 10, 1989) asked for specific recommendations 
regarding CT instruction and assessment, and offered a revision of the general 
statement on teaching and assessinga cognitive skill. Round 5 included several 
quotations culled from the panelists’ earlier responses and invited comments and 
reactions. 


The experts’ comments regarding the various quotations included in each round 
added greatly to the project director's understanding of the experts’ overall views. From 
these and the responses to specific Round 5A, 5B and 5C questions, the project director 
assembled a draft report of all Delphi findings, including recommendations. Round 6, 
(Sept. 25, 1989) circulated that draft and gave the CT experts the opportunity to express 
their views or make comments for inclusion in the final report, which went through its last 
revisions in Nov. 1989. 
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ILL -- The Cognitive Skill Dimension of Critical Thinking 


FINDING: As indicated in Table 1, the experts find good critical 
thinking to include both a skill dimension and a dispositional 
dimension. The experts find CT to include cognitive skills in (1) 
interpretation, (2) analysis, (3) evaluation, (4) inference, (5) 
explanation and (4) self-regulation. Each of these six is at the core 
of CT. Associated with each are criteria by which its execution can be 
meaningfully evaluated. However, no attempt is made here to specify 
those criteria since ample critericlogical discussions exist in the 
literature. 


Concerned not to generate misunderstandings, the experts offer many 
cautions about the analysis of CT in terms of skills and sub-skills. The 
experts warn that good CT is not rote, mechanical, unreflective, 
disconnected execution of sundry cognitive processes. They caution not 
to lose sight of the whole while attempting to attend well to its many 
parts. 

RECOMMENDATION 1: All CT instruction should aim at deveicping good 
critical thinkers -~ persons who can integrate successful execution of 
various skills in the CT enhanced classroom with the confidence, 
inclination and goad judgment to use these powerful tools in their 
other studies and in their everyday lives. Persons who have 
proficiency in CT skills but fail to use them appropriately are most 
unlikely to be regarded as good critical thinkers. 

RECOMMENDATION 2: Those who seek to infuse CT into the educational 
system to be guided by a holistic conceptualization of what it means to 
be a ood critical thinker. That some aspects of (CT, particularly 
features within its skill dimension, are more readily targeted by 
existing educational assessment strategies should not distort the 
conceptualization of CT nor truncate full-blown CT instruction. 

The experts characterize certain cognitive skills as central or core 
CT skills. The more one achieves proficiency in these skills, the more 
worthy one is of being regarded as adept at CT. The experts are not, 
however, saying that a person must be proficient at every skill to be 
perceived as having CT ability. Considering the panel’s purposes and 


methodology, trying to anelyze CT in terms of necessary and sufficient 


conditions would have haJ strong negative utility. Thus, in view of the 
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precision which the question permits, the panel, early in the Delphi 
process, decided to strive for a consensus on the core skills. The panel 
was not asked to name skills without which a persan is surely not a 


critical thinker. 


Responses to Rounds 4 and SA reveal the experts to be virtually 
unanimous (No9SZ) on iectluding analysis, evaluation, and inference as 
cem ral to CT. But in response to Round 6 one assessment expert strongly 
dissented regarding the inclusion of interpretation, arguing that it was 
properly a part of communication, not CT. The same expert noted that 
analysis, as defined in this report, overlaps with reading and listening. 
These points raise obvious difficulties for CT assessment, particularly 
as one attempts to make finer differentiations between CT and 
communication or between analysis—-in-the-CT-sense and analysis—in-the- 
ceading-sense. Regarding self-regulation the expert said, "I think this 
is where testing must merge with teaching." In response to Round ó 
another assessment espert pointed out that, as compared to the others, 
self-regulation appears to be a skill of a different kind or level. In 
self-regulation one applies the other CT skills to one’s own CT, by, for 
example, evaluating ore’s own inferences. This gives CT an interestingly 
recersive Character. However, as this expert noted, the meta-cognitive 
aspect of self-regulation makes it extremely difficult ta assess using 
the standard kinds of paper and pencil instruments. Nonetheless, strang 
consensus (N 8759 exists that interpretation, explanation and self- 
regulation are ventral to CT. (Far detailed results see the response 


tables un page 10 of the Delphi letter for Round SA in Appendix C.J 


fuer. 
Ww 


RERESET I a Pe Taea bet nt aE Tae GP Bee 
s i x - 





; FINDING: There is consensus that one might improve one's own CT in 
n several ways. The experts agree that one could critically cvamine and 
evaluate one's own reasoning processes. One could learn hov to think 

more objectively and logically. One could expand one's repertoire of 

those more specialized procedures and criteria used in ditferent areas 

of human thought and inquiry. One could increase one's base of 

information and life experience. 

It was readily apparent that the experts do not regard CT as a body 
of knowledge to be delivered to students as one more school subject along 
with others. The panel sees CT, like reading and writing, as having 
applications in all areas of life and learning. And, as several pointed 
out, CT instruction, like reading and writing, can occur in programs rich 
with discipline-specific content or in programs which rely on the events 
in everyday life as the basis for developing one’s CT. 

FINDING One implication the experts draw from their analysis of CT 

skills is this: "while CT skills themselves transcend specific subjects 

or disciplines, exercising them successfully in certain contexts 

demands domain-specific knowledge, some of which may concern specific 

methods and techniques used to make reasonable judgments in those 
specific contexts." 

Although the identification and analysis of CT skills transcend, in 
significant ways, specific subjects or disciplines, learning and applying 
these skills in many contexts requires domain-specific knowledge. This 
domain-specific knowledge includes understanding methodological 
principles and competence to engage in norm-regulated practices that are 
at the core of reasonable judgments in those specific-contexts. The 
explicit mention of "evidential, conceptual, methodological, 
writeriological, or contextual" considerations in connection with 
explanation reinfurces this point. Too much of value is lost if CT is 
conceived of simply as a list of logical operations and domain-specific 


knowledge is conceived of simply as an aggregation of information. 


Inquiry into the nexus of reasonable judgment and actual application can 
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produce new appreciations of the necessity of robust concepts of both CT 
and domain-specific knowledge in education. 

RECOMMENDATION 3: Since becoming adept at CT involves learning to use 

CT skills effectively in many different contexts, the experts insist 

that "one cannot overemphasize the value of a solid liberal education 

to supplement the honing of one’s CT skills and the cultivating of 

one’s CT dispositions.” 

The experts caution that CT skills can usefully be grouped and sub- 
classified in a number of legitimate ways. Hence, the sub-classification 
which resulted from this Delphi research should not be interpreted as 
necessarily excluding all others. Indeed, while declaring themselves to 
be in agreement with this sub-classification, various participating 
experts have also published their own sub-classifications. There are 
areas of overlap in the classification system which emerged from the 
Delphi research. However, while characterizing each skill and sub-skill 
is important, creating arbitrary differentiations simply ta force each 
and every sub-skill to become conceptually discrete from all the others 
is neither necessary nor useful. In practical contexts the execution of 
sone skills or sub-skills may presuppose others. Thus, order of the 
Delphi listing is not intended to imply the endorsement of any 
psychological, logical or epistemological order or skill-sequence, nor as 
prescribing any educational taxonomy or skill-hierarchy. 

Table 3 lists the skills and sub-skills which the experts identify 
as being at the core of CT. No claim is being made that the list 
exhausts the concept of CT in either breadth or detail. Beyond their 
inclusion in CT, many of the skills and sub-skills identified are 
valuable, if not vital, for other important activities, such as 
communicating effectively. Also CT skills can be applied in concert with 


other technical or interpersonal skills to any number of specific 
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concerns such as programming computers, defending clients, developing a 
winning sales strategy, managing an office, or helping a friend figure 
out what might be wrong with his car. In part this is what the experts 
mean by characterizing these CT skills as pervasive and purposeful. It 
is also fair to say that a particular skill, such as evaluation, or a 
particular sub-skill, such as developing reasons, is essential for 
success in a given endeavor, such as properly diagnosing illness. The 
experts are not concerned that various skills and sut--skills are widely 
used It is not a problem that the skills might be essential elements in 
other endeavors: On the contrary, it would be extremely disconcerting if 
they were not, since the case for infusing CT into the educational system 
depends on CT’s utility across almast all areas of life and learning. 


The experts are clear on the point that not every useful cognitive 






TABLE 3 






CONSENSUS LIST OF CRITICAL THINKING 
COGNITIVE SKILLS AND SUB-SKILLS 






1. Interpretation * Categorization 
* Decoding Significance 
* Clarifying Meaning 

2. Analysis * Examining Ideas 


* Identifying Arguments 
* Analyzing Arguments 








3. Evaluation * Assessing Claims 
* Assessing Arguments 









4. Inference * Querying Evidence 
* Conjecturing Alternatives 
* Drawing Conclusions 






5. Explanation * Stating Results 
* Justifying Procedures 
* Presenting Arguments 







6. Self-Regulation. * Self-examination 
* Self-correction 
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process should be thought of as CT. Not every valuable Lhinking skill is 
CT skill. CT is one among a family of closely related forms of higher- 
order thinking, along with, for example, problem-solving, decision 

making, and creative thinking. Unfortunately the conceptual overlaps and 
complex relationships among all the various forms of higher-order 

thinking have yet to be examined satisfactorily. However, that does not 
imply that one cannot develop a careful and accurate conceptualization of 
the target, CT -- a conceptualization fully adequate to its purpose, 


which is to guide CT assessment and instruction. 





In addition to accord on the listings in Table 3, the Delphi experts 
find remarkable consensus on the descriptions of each of the skills and 
sub-skills. These descriptions are presented in Table 4. The examples 
aseaciated with each sub-skill are intended as clarifications. Some 
readers might see in them suggestions of possible instructional or 
assessment strategies. Others might see in them the tools to initiate 
staff development conversations about the curricular implications. 
However, the panel's consensus has to do with the skill and sub~skill 


descriptions, and does not necessarily extend to the examples. 
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TABLE 4 
CONSENSUS DESCRIPTIONS OF CORE CT SKILLS AND SUB-SKILLS 


1. INTERPRETATION: To comprehend and express the meaning or 
Significance of a wide variety of experiences, situations, data, 
events, judgments, conventions, beliefs, rules, procedures or 
criteria. 


1.1 CATEGORIZATION: 

* to apprehend or appropriately formulate categories, 
distinctions, or frameworks for understanding, describing or 
characterizing information. 

* to describe experiences, situations, beliefs, events, 
etc. so that they take on comprehensible meanings in terms of 
appropriate categorizations, distinctions, or frameworks. 


For example: to recognize a problem and define its character 
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without prejudice to inquiry; to determine a useful way of sorting 
and sub-classifying information; to make an understa:dable report 
of what one experienced in a given situation; to classify data, 
findings or opinions using a given classification schema. 


1.2 DECODING SIGNIFICANCE: 

x to detect, attend to, and describe the informational 
content, affective purport, directive functions, intentions, 
motives, purposes, social significance, values, views, rules, 
procedures, criteria, or inferential relationships expressed in 
convention-based communication systems, such as in language, 
social behaviors, drawings, numbers, graphs, tables, charts, signs 
and symbols. 


For example: to detect and describe a person's purposes in asking 
a given question; to appreciate the significance of a particular 
facial expression or gesture used in a given social situation; to 
discern the use of irony or rhetorical questions in debate; to 
interpret the data displayed or presented using a particular form 
af instrumentation, 


1.3 CLARIFYING MEANING: 

* to paraphrase or make explicit, through stipulation, 
description, analogy or figurative expression, the contextual, 
conventional or intended meanings of words, ideas, concepts, 
statements, behaviors, drawings, numbers, signs, charts, graphs, 
symbols, rules, events or ceremonies. 

* to use stipulation, description, analogy or figurative 
expression to remove confusing, unintended vagueness or ambiguity, 
or to design a reasonable procedure for so doing. 


“or example: to restate what a person said using different words 
or expressions while preserving that person's intended meanings} 
ta find an example which helps explain something to someone; to 
develop a distinction which makes clear a conceptual difference or 
removes a troublesome ambiguity. 


2. ANALYSIS: To identify the intended and actual inferential 
relationships among statements, questions, concepts, descriptions 
or other forms of representation intended to express beliefs, 
judgments, experiences, reasons, information, or opinions. 


2.1 EXAMINING IDEAS: 

* to determine the role various expressions play or are 
intended to play in the context of argument, reasoning or 
persuasion. 
* to define terms. 

* to compare or contrast ideas, concepts, or statements. 

* to identify issues or problems and determine their 
component parts, and also to identify the conceptual relationships 
of those parts to each other and to the whole. 


For example: to identify a phrase intended to trigger a 
sympathetic emotional response which might induce an audience to 
agree with an opinion; to examine closely related proposals 
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regarding a given problem and to determine their points of 
Similarity and divergence; given a complicated assignment, to 
determine how it might be broken up into smailer, more manageable 
tasks; to define an austract concept. 


2.2 DETECTING ARGUMENTS: 

* given a set of statements, descriptions, questions or 
graphic representations, to determine whether or not the set 
expresses, or is intended to express, a reason or reasons in 
support of or contesting some claim, opinion or point of view. 


For example, given a paragraph, determine whether a standard 
reading of that paragraph in the context of how and where it is 
published, would suggest that it presents a claim as well as a 
reason or reasons in support of that claims; given a passage from a 
newspaper editorial, determine if the author of that passage 
intended it as an expression of reasons for or against a given 
claim or opinion; given a commercial announcement, identify any 
claims being advanced. along with the reasons presented in their 
Support, 


2.3 ANALYZING ARGUMENTS: 

* given the expression of a reason or reasons intended 
to support or contest some claim, opinion or point of view, to 
identify and differentiate: (a) the intended main conclusion, (b) 
the premises and reasons advanced in support of the main 
conclusion, (c) further premises and reasons advanced as backup or 
support for those premises and reasons intended as supporting the 
main conclusion, (d) additional unexpressed elements of that 
reasoning, such as intermediary conclusions, unstated assumptions 
or presuppositions, (e) the overall structure of the argument or 
intended chain of reasoning, and (f) any items contained in the 
body of expressions being examined which are not intended to be 
taken as part of the reasoning being expressed or its intended 
background. 


For example: given a brief argument, paragraph-sized argument, or 
a position paper on a controversial social issue, to identify the 
author's chief claim, the reasons and premises the author advances 
on behalf of that claim, the background information used to 
support those reasons or premises, and crucial assumptions 
implicit in the author’s reasoning; given several reasons or 
chains of reasons in support of a particular claim, to develop a 
graphic representation which usefully charecterizes the 
inferential flow of that reasoning. 


3. EVALUATION: To assess the credibility of statements or other 
representations which are accounts or descriptions of a person’s 
perception, experience, situation, judgment, belief, or opinion; 
and to assess the logical strength of the actual or intend 
inferential relationships among statements, descriptions, 
questions or other forms of representation. 


3.1 ASSESSING CLAIMS: 
* to recognize the factors relevant to assessing the 
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degree of credibility to ascribe to a source of information or 
opinion. 

* to assess the contextual relevance of questions, 
information, principles, rules or procedural directions. 

* to assess the acceptability, the level o. confidence 
to place in the probability or truth of any given representation 
of an experience, situation, judgment, belief or opinion. 


For example: ta recognize the factors which make a person a 
credible witness regarding a given event or credible authority on 
a given topics; to determine if a given principle of conduct is 
applicable to deciding what to do in a given situation; to 
determine if a given claim is likely to be true or false based cn 
what one knows or can reasonably find out. 


3.2 ASSESSING ARGUMENTS: 

* to judge whether the assumed acceptability of the 
premises of a given argument justify one’s accepting as true 
(deductively certain), or very probably true (inductively 
justified), the expressed conclusion of that argument. 

* to anticipate or to raise questions or objections, and 
to assess whether these point to significant weakness in the 
argument being evaluated. 

* to determine whether an argument relies on false or 
doubtful assumptions or presuppositions and then to determine how 
crucially these affect its strength. 

* to judge between reasonable and fallacious inferences; 

* to judge the probative strength of an argument’s 
premises and assumptions with a view toward determining the 
acceptability of the argument. 

* to determine and judge the probative strength of an 
argument’s intended or unintended consequences with a view toward 
judging the acceptability of the argument; 

+ to determine the extent to which possible additional 
information might strengthen or weaken an argument. 


For example: given an argument to judge if its conclusion follows 
either with certainty or with a high level of confidence from its 
premises; to check for identifiable formal and informal fallacies; 
given an objection to an argument to evaluate the i:z:cal terce of 
that objection; tc evaluate tne qualis, and applicability of 
analogical arguments; ‘= sudge the logical strength of arguments 
based an “voothetical situations or causal reasoning; ta judge if 
a given argument is relevant or applicable or fas implications for 
the situation at kand; to determine how possible new data might 
lead lagically to the further confirmation or discontirmation gf a 
given opinion, 


4: INFERENCE: To identify and secure elements needed to draw 
reasonable conclusions; to form conjectures and hyrotheses; to 
consider relevant information and to educe the consequences 
flowing from data, statements, principles, evidence, judgments, 
beliefs, opinions, concepts, descriptions, questions, or other 
forms of representation. 
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4.1 QUERYING EVIDENCE: 

* in particular, to recognize premises which require 
support and to formulate a strategy for seeking and gathering 
information which might supply that support. 

x in general, to judge that information relevant to 
deciding the acceptability, plausibility or relative merits of a 
given alternative, question, issue, theory, hypothesis, or 
statement is required, and to determine plausible investigatory 
strategies for acquiring that information. 


For example: when attempting to develop a persuasive argument in 
support of one's opinion, to judge what background information it 
would be useful to have and to develop a plan which will yield a 
clear answer as to whether or not such information is available; 
after judging that certain missing information would be germane in 
determining if a given opinion is more or less reasonable than a 
competing opinion, to plan a search which will reveal if that 
information is available. 


4.2 CONJECTURING ALTERNATIVES: 

* to formulate multiple alternatives for resolving a 
problem, to postulate a series of suppositions regarding a 
question, to project alternative hypotheses regarding an event, to 
develop a variety of different plans to achieve some goal. 

* to draw out presuppositions and project the range of 
E of decisions, positions, policies, theories, 
or beliefs. 


For example: given a problem with technical, ethical or budgetary 
ramifications, to develop a set of options for addressing and 
resolving that problem; given a set of priorities with which one 
may or may not agree, to project the difficulties and the benefits 
which are likely to result if those priorities are adopted in 
decision making. 


4.3 DRAWING CONCLUSIONS: 

* to apply appropriate modes of inference in determining 
what position, opinion or point of view one should take on a given 
matter or issue. 

* given a set of statements, descriptions, questions or 
other forms of representation, to educe, with the proper level of 
logical strength, their inferential relationships and the 
consequences or the presuppositions which they support, warrant, 
imply or entail. 

* to employ successfully various sub-species of 
reasoning, as for example to reason analogically, arithmetically, 
dialectically, scientifically, etc. 

* to determine which of several possible conclusions is 
most strongly warranted or supported by the evidence at hand, or 
which should be rejected or regarded as less plausible by the 
information given. 


For example: ta carry out experiments and to apply appropriate 


statistical inference techniques in order to confirm or disconfirm 
an empirical hypothesis; given a controversial issue to examine 
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informed opinions, consider various opposing views and the reasons 
advanced for them, gather relevant information, and formulate 
one’s own considered opinion regarding that issue; to deduce a 
theorem from axioms using prescribed rules of inference. 


5: EXPLANATION: To state the results of one’s reasoning; to 
justify that reasoning in terms of the evidential, conceptual, 
methodological, criteriological and contextual considerations upon 
which one’s results were based; and to present one’s reasoning in 
the form of cogent arguments. 


5.1 STATING RESULTS: 
* to produce accurate statements, descriptions or 
representations of the results of one’s reasoning activities so as 
to analyze, evaluate, infer from, or monitor those results. 


For example: to state one's reasons for holding a given view; to 
write down for one’s own future use one’s current thinking about 
an important or complex matters to state one's research findings; 
to convey one's analysis and judgment regarding a work of art; to 
state one’s considered opinion on a matter of practical urgency. 


5.2 JUSTIFYING PROCEDURES: 

* to present the evidential, conceptual, methodological, 
criteriological and contextual considerations which one used in 
forming one’s interpretations, analyses, evaluation or inferences, 
so that one might accurately record, evaluate, describe or justify 
those processes to one’s self or to others, or so as to remedy 
perceived deficiencies in the general way one executes those 
processes. 


For example: to keep a log of the steps followed in working 
through a long or difficult problem or scientific procedure; to 
explain one’s choice of a particular statistical test for purposes 
of data analysis; to state the standards one used in evaluating a 
piece of literature; to explain how one understands a key concept 
when conceptual clarity is crucial for further progress on a given 
problem; to show that the prerequisites for the use of a given 
technical methodology have been satisfied; to report the strategy 
used in attempting to make a decision in a reasonable way; to 
design a graphic display which represents the quantitative or 
spatial information used as evidence. 


5.3 PRESENTING ARGUMENTS: 
* to give reasons for accepting some claim. 
* to meet objections to the method, conceptualizations, 
evidence, criteria or contextual appropriateness of inferential, 
analytical or evaluative judgments. 


For example: to write a paper in which one argues for a given 
position or policy; to anticipate and to respond to reasonable 
criticisms one might expect to be raised against one's political 
views; to identify and express evidence and counter~evidence 
intended as a dialectical contribution to one's own or another 
person's thinking on a matter of deep personal concern. 
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6: SELF-REGULATION: Self-consciously to monitor one's cognitive 
activities, the elements used in those activities, and the results 
educed, particularly by applying skills in analysis and evaluation 
to one’s own inferential judgments with a view toward questioning, 
confirming, validating, or correcting either one’s reasoning or 
one’s results. 


6.1 SELF-EXAMINATION: 

x to reflect on one’s own reasoning and verify both the 
results produced and the correct application and execution of the 
cognitive skills involved. 

* to make an objective and thoughtful meta-cognitive 
self-assessment of one’s opinions and reasons for holding them. 

* to judge the extent to which one’s thinking is 
influenced by deficiencies in one’s knowledge, or by stereotypes, 
prejudices, emotions or any other factors which constrain one’s 
objectivity or rationality. 

* to reflect on one’s motivations, values, attitudes and 
interests with a view toward determining that one has endeavored 
to be unbiased, fair-minded, thorough, objective, respectful of 
the truth, reasonable, and rational in coming to one’s analyses, 
interpretations, evaluations, inferences, or expressions. 


For examples to examine one’s views on a controversial issue with 
sensitivity to the possible influences of one’s persona) bias or 
self-interest; ta review one's methodology or calculations with a 
view to detecting mistaken applications or inadvertent errors; to 
reread sources to assure that one has not overlooked important 
information; to identify and review the acceptability of the 
facts, opinions or assumptions one relied on in coming to a given 
point of view; to identify and review one's reasons and reasoning 
processes in coming to a given conclusion. 


6.2 SELF-CORRECTION: 
* where self-examination reveals errors or deficiencies, 
to design reasonable procedures to remedy or correct, if possible, 
those mistakes and their causes. 


Far example: given a methodological mistake or factual deficiency 
in one’s work, to revise that work so as to correct the problem 
and then to determine if the revisions warrant changes in any 
pasition, findings, or opinions based thereon. 
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As is evident, particularly in the descriptions of self-examination 
and self-correction, there are dispositional components to critical 
thinking. Indeed each cognitive skill, if it is to be exercised 
appropriately, can be correlated with the cognitive disposition to do so. 
In each case a person who is proficient in a given skill can be said to 
have the aptitude to execute that skill, even if at a given moment the 
person is not using the skill. But there was a great deal more many 
experts wished say in regard to the personal traits, habits pf mind, 
attitudes or affective dispositions which seem to characterize good 
critical thinkers. 

FINDING: Although the language here is metaphorical, one would find 

the panelists to be in general accord with the view that there is a 

critical spirit, a probing inquisitiveness, a keenness of mind, a 

zealous dedication to reason, and a hunger or eagerness for reliable 

information which good critical thinkers possess but weak critical 
thinkers do not seem to have. As water strengthens a thirsty plant, 

the affective dispositions are necessary for the CT skills identified 

to take root and to flourish in students. 

RECOMMENDATION 4:3 Modeling that critical spirit, awakening and 

nurturing those attitudes in students, exciting those inclinations and 

attempting to determine objectively if they have become genuinely 
integrated with the high quality execution of CT skills are, for the 
majority of panelists, important instructional goals and legitimate 
targets for educational assessment. However, the experts tarbor no 
illusions about the ease of designing appropriate instructional 
programs or assessment tools. 

Procedural, Laudatory and Normative Uses af the Term “CT” 

The experts have a consensus regarding the list of affective 
dispositions which characterize good critical thinkers. This consensus is 
expressed in Table 5. However, whether or not these affective dispositions 


are part of the meaning of "CT" in the way that the cognitive skills are, 


was an issue which divided the experts from the first. It became evident 
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that various experts mean different things when they used the term "CT" in 
reference to its possible disnositional components. 

The ceepest division is between the nearly two-thirds majority who 
hold that the term "CT" includes in its meaning a reference to certain 
affective dispositions and the roughly one-third minority who hold that 
"CT" refers only to cognitive skills and dispositions, but not to affective 
dispositions. The project director put this issue to the panel in several 
different ways, sometimes directly and at other times more obliquely. 
Responses, comments and arguments were shared, as were the objections and 
counter-arguments which they engendered. In the end the panel remained 
divided both numerically and in depth of feeling, with opposing positions 
becoming more strident and entrenched as the debate continued. 

In Round SB, of those expressing an opinion, the majority (41%) 
maintain that the affective dispositions constitute part of the meaning of 
"CT." They argue that these dispositions flow from, and are implied by, 
the very concept of CT, much as the cognitive dispositions are. These 
experts argue that being adept at CT skills but habitually not using them 
appropriately disqualifies one from being called a critical thinker at all. 
Thus, in addition to using "CT" in its procedural sense, these panelists 


also use "CT" in its laudatory sense. They find it sensible to say, "This 


close-minded, unwilling to check the facts and unmoved by reasonable 
arguments that we simply cannot call him a critical thinker." 

The laudatory use of "CT" can suggest approval of how well a person 
applies her CT skills or it can convey praise for the persor. because the 
persan has the proper affective dispositions. While the two-thirds 
majority was eloquent regarding the importance of finding ways to instill 


affective dispositions in students, in the final analysis they were unable 
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to persuade the ather third of their expert colleagues to view these 
dispositions as essential to the concept of CT. The majority was, however, 
persuasive in bringing about virtual unanimity regarding using the 
affective dispositions to describe the paradigm critical thinker. (See 
Table 1.) 
In Round SB a minority (30%) insist on using "CT" in a strict 
pracedural sense, that is as referring only ta a certain judgmental 
process. They distinguish sharply between what is true of critical 
thinking from what is true of good critical thinkers. Their primary 
concern is with the CT skills. They argue that good critical thinkers are 
people who have those skills and certain valuable habits as well. If they 
are goud critical thinkers, then they use their CT skills appropriately 
because good critical thinkers also have some or all of the affective 
dispositions listed in Table 5. But those dispositions are not what is 
meant by "CT." They argue that one would not want to say a sophist is not 
a critical thinker simply because the sophist uses CT skills for deceptive 
or self-interested ends. The sophist, they would maintain, is a critical 
thinker -- but nat an good one (in an ethical sense) The strict 
procerluralists do not find it sensible deny that a person is a critical 
thinker simply because the person, while skilled in CT, fails to check the 
credibility of sources, gives up too soon when asked to work a challenging 
problem, lacks confidence in using reason to approach everyday problems, or 
ignores painful facts. These experts hold that such a person, because af 
nis CT skills, should be called a critical thinker -- but not a good one, 
Gn terms of his effective use of those skills). 
As suggested above, there are two senses of the term "good" which 
' might be operating when one uses the phrase “good critical thinker." One 


sense applies to the thinker’s effectiveness and responds to the question, 
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“How well is this person using CT?" The second sense applies to the 
thinker’s morality and responds to the question, "Is this person's use of 
CT ethical?" In order to clarify which sense of "good" the experts wished 
to convey, Round 4 asked the panel to respond to a proposal that CT might 
have a normative dimension in addition to a skill dimension and a affective 
dispositional dimension. 

FINDING: The mistaken notion that CT has a normative component is 

rejected by the expert panelists. It is an inappropriate use of the 

term to deny that someone is engaged in CT on the grounds that one 

disapproves ethically of what the person is doing. What “CT” means, 

why it is of value, and the ethics of its use are best regarded as 

three distinct concerns. 

The majority of experts (52%) forcefully reject the proposed 
nernative use of "CT." They hold that it is one thing to say what 
something is, and another thing to say how it ought to be used. A 
person's skilis and attitudes are what they are, even if the person 
suffers from certain ethical inadequacies. 

Only a small group (7%), argue in favor of using "CT" in a 
normative sense. This minority of experts, all of whom also use "CT" in 
its commonly understood laudatory sense, hold that the true meaning of 
"CT" extends to a certain set of ethical norms and social values. For 
example, they would be willing ta say that a defense attorney who uses CT 
skills to cause a mistrial or win acquittal for a guilty client ought not 
be dignified with the title of critical thinker. By the same taken, the 
prosecutor who uses CT skills to contrive a way to mislead a gullible 
Jury into convicting and punishing an innocent person is not a critical 
thinker. Since neither sufficiently value truth and since both appear to 
lack the moral fiber to eschew deliberate deception in the practice of 


their socially important professions, neither attorney should be accorded 








the moral approval which calling them critical thinkers would imply. 

The debate turned aut toa be instructive in another way as well. The 
consensus (74% in support and 4% opposed) was that this report should 
express the experts’ fullest support and appreciation of the immense 
personal and social importance of CT. 

The panel shares a solid consensus about the importance of CT as a 
tool of inquiry, as a liberating force in education, as a powerful 
resource in one’s personal life, and as a vital component in a rational 
democratic society. It is extremely unlikely that any panelist would 
condone using CT for inmoral, deceptive, or unjust purposes. However the 
personal and civic value of CT and sensitivity to the morality of its use 
are nat acceptable grounds for building a normative dimension into the 
meaning of the term "CT". Some even saw such an effort as misguided and 
potentially destructive of the CT movement. Giving "CT" a normative 
twist could, they argue, lead to unwarranted limitations on open inquiry 
and to unjustifiable ideological restrictions on the very concept of 
being a "thinking" person. The totalitarian specter this conjures up is 


the antithesis of the liberating critical spirit described earlier. 


Dispositions of the Good Critical Thinker 


FINDING: To the experts, a good critical thinker, the paradigm case, 
is habitually disposed to engage in, and to encourage others to engage 
in, critical judgment. She is able to make such judgments in a wide 
range of contexts and for a wide variety of purposes. Although perhaps 
not always uppermost in mind, the rational justification tor 
cultivating those affective dispositions which characterize the 
paradigm critical thinker are soundly grounded in CT's personal and 
civic value. CT is known to contribute to the fair-minded analysis and 
resolution of questions. CT is a powerful tool in the search for 
knowledge. CT can help people overcome the blind, sophistic, or 
irrational defense of intellectually defective or biased opinions. CT 
promotes rational autonomy, intellectual freedom and the objective, 
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reasoned and evidence-based investigation of a very wide range of 
personal and social issues and concerns. 


Thus, in addition to possessing CT skills the good critical thinker 
can be characterized by certain affective dispositions or habits of mind. 
These dispositions, listed in Table 5 below, flow from two sources: 
characteristics which the experts judge to hold true of good critical 
thinkers, and the affective dispositions the experts judge to be part of 
cT in its fullest realization. The majority of the experts (61%) regard 
the dispositions listed in Table 5 as part of the conceptualization of 
CT. A consensus exists (83% in favor) that good critical thinkers can be 


characterized as exhibiting these dispositions. 


TABLE 5 
AFFECTIVE DISPOSITIONS OF CRITICAL THINKING 


APPROACHES TO LIFE AND LIVING IN GENBRAL: 

* inquisitiveness with regard to a wide range of issues, 

* concern to become and remain generally well-informed, 

* alertness to opportunities to use CT, 

* trust in the processes of reasoned inquiry, 

* self-confidence in one’s own ability to reason, 

* open-mindedness regarding divergent world views, 

* flexibility in considering alternatives and opinions, 

* understanding of the opinions of other people, 

* fair-mindedness in appraising reasoning, 

* honesty in facing one’s own biases, prejudices, 
stereotypes, egocentric or sociocentric tendencies, 

* prudence in suspending, making or altering judgments, 

* willingness to reconsider and revise views where honest 
reflection suggests that change is warranted. 


APPROACHES TO SPECIFIC ISSUES, QUESTIONS OR PROBLEMS: 
* clarity in stating the question or concern, 
* orderliness in working with complexity, 
* diligence in seeking relevant information, 
* regsonableness in selecting and applying criteria, 
* care in focusing attention on the concern at hand, 
* persistence though difficulties are encountered, 
* precision to the degree permitted by subject and circumstances. 
95 
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The experts are not saying that a person whose metaphysical, 
epistemological, political, cultural or religious view of the world is 
different from one’s awn is, ipso facto, not a good critical thinker. 
Beliefs are not atoms each of which is at any moment subject to being 
reconsidered independently. Beliets form intricately interconnected 
systems of thought. To focus critical attention on any of them, 
particularly those more central or fundamental to one’s own view of the 
world, can cause reverberations throughout one’s entire belief system. 
Thus in advocating CT the panelists are not urging ideological 
conformity. Indeed, just as many experts argued that an over-emphasis on 
the values of CT could lead ta trouble, others warn that an over-emphasis 
oan the skills dimension of CT to the exclusion of the affective 
dispositions might have the unfortunate result of making some students 
close~minded, intellectually inflexible and dogmatic. 

RECOMMENDATION 9: Just as with the cognitive dimension of CT, when 
conceiving of the education or assessment of critical thinkers, it is 
important to consider ways of developing materials, pedagogies, and 
assessment tools that are effective and equitable in their focus on 
these affective dispositions. The cultivation of these dispositions is 
particularly important to insure the use of CT skills outside the 
narrow instructional setting. Persons who have developed these 
affective dispositions are much more likely to apply their CT skills 
appropriately in both their personal life and their civic life than are 
those who have mastered the skills but are not disposed to use them. 

AS with the listing of cognitive skills earlier, the panel does not 
intend that each disposition be considered a necessary condition. The 
experts are characterizing the ideal. In setting forth the concept of 
the paradigm critical thinker, they intend to express a goal toward which 
all might strive. These virtues require a measure of maturity arid 


personal development not commonly found in college sophomores or twelfth 


graders. Yet ta delay embarking cn the practices and disciplines which 
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will lead to these virtues would be an even more profound mistake. 


RECOMMENDATION 46: From early childhood people should be taught, for 
example, to reason, ta seek relevant facts, to consider options, and to 
understand the views of others. It is neither impractical nor 
unreasonable to demand that the educational system teach young people 
the habits of mind which characterize the good critical thinker, 
reinforce those practices, and move students well down the path toward 
their attainment. 


V -- Further Recommendations on CT Instruction and Assessment 


Several pedagogical and assessment implications follow from the 
dispositional dimension of CT, implications which might not be apparent 
if educators focused only on the skill dimension of CT. The education of 
good critical thinkers is more than training students to execute a set of 
cognitive skills. For example, in terms of pedagogy, modeling how to 
evaluate critically that information which students would normally accept 
uncritically and encouraging them to do the same can do wonders for 
developing their confidence in their CT ability. With this confidence 
students are much more likely to try thinking for themselves. Just as 
instruction should not focus on skills only, assessment which focus on 
skills only may give a misleading or incomplete picture of someone's 


strengths as a critical thinker. 


The CT Goal 


RECOMMENDATION 7: Because CT helps students with a wide range of 
educational, personal and civic concerns in a rational way, the 
academic goal of CT instruction, regardless of the educational level, 
should be furthering students in the development of their CT cognitive 
skills and affective dispositions. 
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TABLE 6 
CONSENSUS STATEMENT ON TEACHING AND ASSESSING CT SKILLS 










A CT skill, like any skill, is the ability to engage in an activity, process or 
procedure. In general, having a skill includes being able to do the right thing at the 
right time. So, being skilled at CT involves knowing, perhaps implicitly or without the 
ability to articulate this knowledge, both a set of procedures and when to apply those 
procedures. Being skilled also involves having some degree of proficiency in executing 
those procedures and being willing to do so when appropriate. Reflecting on and 
improving one’s CT skills involves judging when one is or is not performing well, or as 
well as possible, and considering ways of improving one’s performance. Learning CT 
involves acquiring the ability to make such self-reflective judgments. 











Skills, particularly CT cognitive skills, can be taught in a variety of ways, such 
as by making the procedures explicit, describing how they are to be applied and 
executed, explaining and modeling their correct use, and justifying their application. 
Teaching cognitive skills also involves exposing learners to situations where there are 
good reasons to exercise the desired _ procedures, judging their performance, and 
providing the learners with constructive feedback regarding both their proficiency and 
ways tcimprove it. Instruction might start with situations that are artificially simple, 
but should culminate in situations that are realistically complex. Particularly in the case 
of CT, the learners must contribute a solid measure of personal effort, attention, practice, 
desire, and, as they learn how, self-monitoring. Teaching skills involves motivating 
learners to achieve higher levels of proficiency and, particularly in the case of CT, 
independence. It also involves coaching learners on how they can achieve those goals. 
















In theory there are several ways persons can be judged to be more or less 
proficient in a given CT skill or at the integrated use of related CT skills. One way 
is to observe a person over time performing those activities, processes or procedures 
generally regarded as presupposing that skill for proper execution. One then mah... « 
judgment regarding the degree to which tlie person possesses the general skill in 
question. Asecond way is to compare the outcomes (if any) that result from executing 
a given skill against some set of criteria. A third way is to query persons and receive 
their descriptions of the procedures and judgments they are using as they exercise that 
skill, would use if they were to perform that skill, or did use when they performed that 
skill. A fourth way is to compare the outcomes (if any) that result from performing 
another task ageinst some set of criteria, where the performance of that other task 
has been shown tocorrelate strongly with exercising the skill of interest. However, 
that such correlations exist between any other task and CT, or any of its sub-skills, has 

yet to be established in the research literature. 






















Each of the four ways of CT assessment has limitations as well as strengths. No 
matter which ways are used, it is important to ensure that the assessment conditions 
foster an attitude in which the subjects are disposed to use their skills as well as they 
can, and ate not constrained or inhibited from doing so. In our view it is highly 
advantageous to gather evidence regarding CT performance in many situations, using 
several assessment methods, so as to compile a composite picture of the subject and to 
cross check the results of any one way of assessment. 
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Either to transform CT into one subject field among others, or to 
narrow the range of CT applications strictly to domain-specific subject 
content, would be to truncate its utility, misapprehend its nature and 
diminish its value. Within the overall curriculum the goal of learning 
CT can be clearly distinguished from the goal of learning domain-specific 
content. And yet, while these two goals can be distinguished, the 
experts do not wish to deny one of the best ways to learn CT is within a 
subject context. 

RECOMMENDATION 8: Direct instruction in CT and assessment of CT should 
be an explicit parts of any course granted approval for purposes of 
satisfying CT requirements, whether that course is a CT course per se 
or a course in a given subject field. The primary academic criterion 

in the evaluation of a proposed instructional program for purposes of 

achieving the CT goal should be whether the program will further the 

development of students’ CT skills and dispositions. 
The CT Curriculum 

Given that CT has, in many cases, become a college general education 
requirement, secondary schools can be expected to begin to develop 
college preparatory CT programs. However, the value of CT extends well 
beyond its importance as a university-level inquiry tool. CT is vitally 
important in the personal and civic life of all members of society. A 
significant percentage of the citizenry will not graduate from high 
school, or if they graduate, will not have the benefit of post-secondary 
education. 

RECOMMENDATION 9: Thus, CT instruction should not be reserved only for 
those who plan to attend college. Wor should it be deferred until 
college, since it is not likely to be etfective if it were. 

RECOMMENDATION tò: Explicit attention to the fostering of CT skills and 
dispositions should be made an instructional goal at all levels of the 
K-12 curriculum. The cultivation of CT dispositions and an insistence 


on giving and evaluating reasons, should be an integral part of 
elementary school education. In middle schools and high schools, 
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instruction on various aspects and applications of CT should be 

integrated into all subject area instruction. Specific courses in CT 

and an advanced placement examination program in CT for college bound 

students should be developed. Althaugh for gnod reasons at the post- 

secondary level CT programs are generally associated with departments 

of philosophy, no academic unit should be restricted in principle from 

participating in an institution's CT program, provided that the overall 

institutional program in CT equips students to apply CT to a broad 

range of educational, personal and civic subjects, issues and problems. 

There is growing evidence of the successes, both scientific and 
economic, of those industrialized democracies which emphasize demanding 
academic assessment and set firm educational standards for career and 
profassional advancement. Assessment that counts is unquestionably a key 
factor in pramoting academic achievement. 

RECOMMENDATION 11: Thus, minimum CT proficiency expectations should be 

set for each educational level, including promotion in grade, high 

school graduation, college entrance, and graduate school admission. 

The CT Assessaent 

The development of valid and reliable assessment strategies from 
which teachers can draw reasonable inferences about students’ CT, in 
contrast to their domain-specific knowledge or other academic abilities 
(such as reading or writing), is essential. CT assessment strategies, 
whether for use in the individual classroom or for broader purposes, must 
not simply reward arriving at correct answers. They must, however 
recognize achieving correct answers by way of good CT. The challenge of 
CT assessment is not to let what is easily measured restrict our sense of 
the fullness of CT. It would be shameful if those assessment instruments 
which focus only on CT skills drove our CT curricular design and caused 
the dispositional conpanents of goad CT to be neglected. 

RECOMMENDATION 12: In evaluating the acceptability of a CT assessment 


strategy or instrument one should consider content validity, construct 
validity, reliability, and fairness. 
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(D Content Validity: The strategy or instrument should be based 
on an appropriate conceptualization of CT and a clear understanding of 
which aspects of CT the assessment targets. Each task or question should 
be evaluated to insure that correctly responding to that item is not a 
matter of rote learning or information recall. Whether for the classroom 
or for broader educational purposes, CT assessment should include 
strategies for targeting CT's dispositional dimension as well as its 
cegnitive skills dimension. 

| 2) Construct Validity: In acceptable CT assessment each task or 
question should have been evaluated to insure that students who answer 
correctly do so on the basis of good CT and that inadequate or wrong 
responses are the result of weak or inadequate CT. Entire strategies or 
specific items on which good CT leads to wrong answers, or poor CT to 
right answers, should not be used. 

(3) Reliability: In acceptable CT assessment each task or question 
should have been evaluated to insure that good critical thinkers 
generally do better on that item than weak critical thinkers. If 
different persons are involved in evaluating the results, for example 
grading essays or judging presentations, the evaluations of the different 
judges should be cross-checked to assure that their findings are 
reliable, that is, generally consistent with one another. However, it is 
an open question whether the levels of achievement associated with the 
different CT sub-skills and affective dispositions are positively 
correlated. Empirical research on how the sub-skills correlate with each 
other and with various dispositions has yee tö be undertaken. Thus, at 
this time, due caution should be exercised regarding how to interpret 


technical measures of test-form reliability in the case of paper and 
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pencil CT assessment instruments. 

(4) Fairness: CT assessment should not unfairly disadvantage or 
advantage groups of students on the grounds of reading ability, domain- 
specific knowledge Cbroadly understood as including the evidential, 
conceptual, methodological, criteriological, contextual considerations, 
or familiarity with technical vocabulary], gender or age related life 
experience, ethnicity or socio-economic status, differences in social 
norms, or differences in cultural assumptions. CT assessment locates CT 
tasks and questions in some assumed context, either subject-specific, 
everyday life, or fictional. Thus, guaranteeing that all students, 
regardless of their individual backgrounds, will come to the CT 
assessment on a perfectly equal basis in terms background knowledge, 
reading ability, life experiences, etc. is impossible. However, 
examining the assessment strategy or instrument to be sure that these 
factors do not unfarrly influence the results is prudent and reasonable. 
Although ane cannot eliminate the influence of these variables, one may 
be able to neutralize or control for their affects. 

Tie fairness criterion applies both to discipline-neutral and 
discipline-specific CT assessment. Within curricular programs 
discipline-specific CT assessment is encouraged, since it is possible for 
ane to be fair in one’s presumptions regarding subject-specific criteria, 
concepts, methodologies, evidence, information and terminology. The 
challenge of such assessment is to factor out the discipline content in 
order to access the strength or weakness of the CT. It is worth noting 
that discipline-neutral CT assessment also makes similar assumptions 
regarding the everyday contexts which form its topic content. 


RECOMMENDATION 13: CT assessment should occur frequently, and it should 
be used diagnostically as well as summatively. Different kinds of 
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variety of contexts. 


which is supportive of CT. 


contrasts in how they executed various CT tasks. 


instruments should be employed, depending on which aspect of CT is 
being targeted and where students are in their learning -- the 
introductory stage, the practice stage, the integration stage or the 
generalized transfer stage. Although the veteran CT instructor is able 
to assess studsnts continuously, CT assessment should be made explicit 
to reinforce its worth in the eyes of the students, their families, and 
the public. It should be made explicit to support the goals of 
educators seeking to improve the curriculum. And it should be made 
explicit to properly inform educational policy formation. 


The CT Instructor 


RECOMMENDATION 14: Teaching CT is most effective if the instructor 
models CT dispositions and the proper use of CT skills in the very 
process of instruct.on. Regardless of the subject area, students 
should be encouraged to be curious, to raise objections, ask questions, 
point out difficulties in the instructor's position. These objections 
and questions should be clarified, interpreted, and examined 
objectively. Students should be given reasons for doing things a 
certain way, rather than being dogmatically told how to do them. 
Instruction should bridge the gap between the subject and the student's 
own experience. In the case of CT instruction, the topics af 
discussion should not be restricted to factual matters or academic 
subjects, but should include issues which have normative, moral, 
ethical or public policy dimensions. 


The ideaZ CT instructor will integrate instruction in CT in a 


variety of subject areas. She will teach specific CT skills directly 


using these subjects as content for the application of those skills. 


interactions with colleagues. She will provide her students with 


thought-provoking subjects to learn about, and projects to undertake. 


an, articulate, share and discuss justifications, explanations and 


student's progress, achievement or proficiency in CT continuously. 


RECOMMENDATION 15: For CT to infuse the K-12 and college curriculum, 
teacher "training" should give way to teacher "education." If teachers 
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She will model CT in her teaching and her 


She 


will help students elaborate, transfer and generalize these skills to a 


She will create a classroom and school environment 


She will engage students in social activities requiring them to reflect 


She will evaluate each 





ere to model (CT, so must those who have an instructional role in 
teacher preparation or staff development. In all instruction, and 
particularly in CT instruction, both faculty and leaders of faculty 
development should model CT. They should foster the students’ 
confidence in their own powers of reason, rather than dependency on 
rote learning. They should nurture in students open-mindedness, 
attention to alternatives, and as much precision of thought as the 
subject and circumstances permit. 


YI_-- The Delphi Research Panel 


The Delphi research findings reported here result from the 
participation of forty-six scholars, educators and leading figures in CT 
theory and CT assessment research. Roughly half the panelists are 
Primarily affiliated academically with Philosophy (52%), the others are 
affiliated with Education (22%), the Social Sciences (20%), or the 
Fhysical Sciences (4%). 

It would be a mistake to construe participation in this research 
project as implying that a person agrees with all the findings. Thus, 
where consensus is reported a minority of panelists hold divergent views. 
Where mear unanimity is reported a some panelists may may not be in full 
accord with how the specifics are expressed. In the end, however, after 
reviewing the draft Delphi findings presented in Round ó, only one of the 
forty-six explicitly opted to be listed as a participant only, but mot as 
supporting the document. 

These Delphi findings fix an important moment in time. It is a 
moment when the efforts of forty-six experts possessing special 
experience and knowledge in matters relating ta CT converged with a view 
toward discovering if some measure of general accord could be found. As 
we mave from the successes of the eighties inta the decade of the 
nineties, the persons who participated in this project hope that the 
findings of expert consensus reported herein will advance critical 


thinking and help shape the future of CT instruction and CT assessment. 
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Delphi Research Project on CT Assessment 


The Education Testing Service (ETS) Academic Profile Test measures 
the academic abilities of CT, reading, writing, and using mathematical 
data, all within the context of three major academic areas —- 
humanities, social sciences and natural sciences. This is a multiple 
choice instrument with an optional critical essay (locally scored). It 
assumes that students have completed most or all of their general 


education -- that is, completed the sophomore college year. 


ETS also has sections targeting logical reasoning and analytical 
thinking on several of its widely used instruments such as the (a) Law 
School Admissions Test, (LSAT) "Logical Reasoning" section, (b) Graduate 
Record Examination, (GRE) -- General Test, the Analytical section, (c) 
Advancea Flacement Test, subject-matter based CT questions, (d) National 
Assessment of Educational Progress, higher order thinking and 
laboratory-based questions, (e) Foreign Service Test, in-basket portion, 
and (f) Graduate Record Examination -- Advanced Test in Philosophy 1972- 
1982. ETS is adding a section of Critical Reasoning Questians to the 
Graduate Management Admissions Test (GMAT). ETS has constructed 
branching tests of infoarmation-seeking and decision-making, some paper- 
and-pencil ancl some on computers. An example is the clinical practice 


test prepared for the National Board of Respiratory Care, Shawnee 
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Mission, Kansas. 


Stephen Norris and R. King, through the Institute for Educational 
Research and Development at Memorial University of Newfoundland, has 
developed the Test on Appraising Observations 1983. Using the backdrop 
of a common but fictional situation subjects are asked to judge the 


relative credibility of the claims made by various characters. 


The American College Testing Program (ACT) in 1988 produced the 
“Collegiate Assessment of Academic Proficiency" (CAAF). ACT describes 
CAAF is designed to measure selected academic skills including reading, 
writing, mathematics, CT, and scientific reasoning. Tne CT Test 
measures the ability to "clarify, analyze, evaluate, an. extend 
arguments." The test is composed of passages commonly encountered in 


postsecondary curriculum followed by multiple choice test questions. 


Faul M. Ramirez discusses the "Valett Inventory of CT Abilities" 


(VICTA) in The Reading Teacher. vol. 41, Dec. 1987, page 348. 


THE NINTH MENTAL MEASUREMENTS YEARBOOK, (NMMY), lists commercially 








available tests in print along with reviews and research data. Many of 
these are also described ad reviewed by Norris and Ennis in their 


useful Evaluating CT, Midwest Fublications, Pacific Stove, CA, 19789. 


#249 “Cornell Critical Thinking Test, Levels X and Z." ‘Ennis, Millman, 
Tomko 1761-1982, Midwest Publications, Pacific Grdve, CA. 
Reviewed in Educational and Psychological Measurements 1983 


Vol. 43.5 pp. 1187-1197, by Medjeski and Michael. 
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#370 "Ennis-Weir Argumentation Test, Level X: An Essay Test of Rational 
Thinking Ability," (Robert Ennis and Eric Weir) 1982, Illinois 
Thinking Froject, University of Illinois, Wrbana, IL. 

Reviewed by Herbert Rudman, Michigan State, in NMMY. 

#391 “Ennis-Weir Critical Thinking Essay Tests An Instrument for 
Testing/Teaching."” (Robert Ennis and Eric Weir) 1983, Midwest 
Publications, Pacific Grove, CA. 

#1347 "Watson-Glaser Critical Thinking Appraisal" 1942-80. Described 
and reviewed by two persons in the NMMY, many citations of 
other research regarding this instrument. 

#751 "New Jersey Test of Reasoning Skills," 1983, Virginia Shipman, 
Institute for the Advancement of Philosophy for Children. 

#1258 "Test of Inquiry Skills" 1979, Australian Council for Educational 
Research. For junior high grades, this test purports to 
evaluate a range of research, study and critical thinking 
skills in the sciences. 

#1061 "Rass Test of Higher Cognitive Frocesses" (John Ross and Catherine 
Ross) 1976-79, Academic Therapy Publications. For grades 4-4, 
this test includes sub-scores on analogies, deductive 
reasoning, missing prewises, questioning strategies, and 
relevance of information. 

#1248 “Test of Cognitive Skills" 1981, McGraw Hill. For grade levels 2- 
12, this test includes sub-scores on sequencing, analogies, 
menory, and verbal reasoning. 

#122 "Basic Skills Assessment" 1977-81, McGraw Hill. Included in the 
teading package is a sub-score on inference and evaluation. 
In the writing package is a sub-score on logical evaluation. 


#1269 “Test of Problem Solving" 1984, LinguiSystem Inc. For ages 6-12, 
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this tests a child's thinking and reasoning abilities critical 
to events of everyday life. It includes sub-scores on 
explaining inferences, determining causes, negative why 
questions, etc. 

#272 "Corrective Reading Mastery Test" 1980, Science Research 
Associates, Inc. Designed to measure the effectiveness of 
corrective reading programs, this test includes sub-scores on 
deductions, classifications, analogies, inductions, statement 
inference, hypothesis/evidence. 

#1302 “Deductive Reasoning Test" (J. M. Verster) 1972-73, National 
Institute for Personnel Research, South Africa. Focuses on 
syllogistic problems and designed for for candidates for 
graduate scientists and higher professions. 

#1010 "PSI Basic Skills Test for Business and Industry" 1981-1982, 
Psychological Services Inc. Includes sub-scores on problem 
solving, decision making, reasoning and classifying. 

#106 “Ball Aptitude Battery" the Ball Foundation. Used to tests nersons 
for occupational placements, this instrument includes sub- 
scores on inductive reasoning, analytical reasoning, idea 


fluency, and shape assembly. 
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APPENDIX C 
The Delphi Research Letters 


Over course of this research sixteen “Delphi Letters" were 
sent to the experts participating in the APA Committee on Pre- 
College Philosophy Delphi Research Project on CT assessment. 
Hight letters constituted the specific Citasaseive Delphi 
research rounds. The eight others were for purposes of planning, 


clarifying procedures, providing information, and sustaining 


involvement. 
Feb. 11, 1988 ROUND 1 
Mar. 1, 1988 Information and Plans 
Mar. 14, 1988 ROUND 2 
Apr. 14, 1988 Plans and Procedures 
May 4, 1988 ROUND 3 
May 18, 1988 Procedures and Involvement 
June 28, 1988 Information and Involvement 
Sept. 1, 1988 | Procedures and Information 
Sept. 23, 1988 ROUND 4 
Nov. 22, 1988 Plans and Involvement 
Feb. 7, 1989 Information and Involvement 
Feb. 28, 1989 ROUND 5A 
Mar. 6, 1989 ROUND 5B 
Mar. 10, 1989 ROUND 5C 
May 9, 1989 Information and Involvement 
Sept. 25, 1989 ROUND 6 
Oe 
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Ch Callfornia State University, Fullerton 
Fullerton, California 92634 





Department of Philosophy Feb. 11, 1988 
(714) 773-3611 Rounp 1 


Dear Colleague, 


Gary Matthews, Chair of the APA Committee on Pre-College 
Philosophy, asked me to head up an ad hoc committee on testing 
critical thinking. To get the project rolling. he suggested severa 
names of people interested in the question of haw to validly and 
reliably test critical thinking skills. I adaed a few more. Here are 
some of the particulars. 


1) By using a modified Delphi approach, I think the necessity for 
actual meetings can be largely, if not entirely eliminated. Committee 
members' contributions will involve sending their reasoned and timely 
responses to questions, given their particular background and 
expertise. 


2) There are, no doubt, a great number of other people who are 
interested in the question of testing critical thinking and who have 
valuable expertise which would heip us in in dealing with this 
question. They should also be invited to participate. 


3) Since our charge is rather vague, I propose that among the 
first things we should do is agree on priorities. To do this, using 
the Delphi process, let me lead off with some assumptions and 
questions: 


First assumption: Most of the members of our group will come at 
the issue of testing critical thinking with the orientation of 
philosophers or logicians who teach at the post:-secondary level, 
rather than as K-12 educators, psychologists, or personnel 
directors (all of whom also have legitimate theoretical and 
practical interests in assessing critical thinking). Given the 
interests of the American Philosophical Association, this js 
acceptable. 


Second assumption: Critical thinking can be defined Operationally 
to the extent that it can become a dependent variable in a valid 
and reliable assessment tool. 


QUESTIONS: 
1. Do you agree with the two assumptions? How would you 


amerd/clarify them? Why? 


2. To which educational level (from Kindergarten through post- 
Baccalaureate) should the committee cive priority? Why so? 


3. After looking the attached preliminary list, whom else 
would you recommend be added to our committee? 


The Cat:torma State University 
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For the Delphi process, which is very interactive, to function 
optimally, reasonably quick turn around is needed. Let's target a 
Feb. 29 postmark. Is that possible for you? 


Please send responses, suggestions, comments, etc. to: 


Ad Hoc APA Committee on Critical Thinking Testing 
c/o Dr. Peter A. Facione 

Professor of Philosophy and Education 

Dept. of Philosophy 

California State University, Fullerton 

Fullerton, CA 92634 


Like math and composition, at many colleges throughout the 
country CT is being built into the curriculum. For example, the 
California State University system, which enrolls hundreds of 
thousands of undergraduates, has implemented a system-wide critical 
thinking requirement as part of its general education package. If 
those of us who teach critical thinking were able to agree on a way or 
ways it could ke tested, what a positive contribution that could be to 
the quality of that curriculum. 


I sincerely hope you will agree to become an active participant 
in:what promises to be a most. interesting and important effort. 


I've included some reference material from The Ninth Mental 
Measurements Yearbook regarding published instruments which purport to 
mleaSure critical thinking and/or related cognitive skills. 


F Yours3sincerely, 


Pete Facione 
cc. Gary Matthews, University of Massachusetts, Amherst 


Steve Tigner, University of Toledo, Ohio 


ATTCHMENTS: Preliminary List of Committee Members 
Quick Survey of Published Instruments 


TO BE DEVELOFED: Bibliography on Testing Critical Thinking 
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California State University, Fullerton 
Fullerton, California 92634 


SF 





Department of Philosophy 
(714) 773-3611 


March 1, 1988 
Dear Colleague, 
Early sesponses to the first Delphi round are very encouraging! 


Some of you know what I'm talking about, others, new to the effort, 
need to be brought up to speed. Let me back track a bit and explain. 


I am writing to you because you were nominated by one or more of 
your colleagues as a person interested in the question of testing 
critical thinking. You are being invited to participate in the work of 
an ad hoc American Philosophical Association subcommittee concerned with 
the problem of testing critical thinking. In its boldest form, our ain 
is to find ways to validly and reliably test critical thinking, or find 
solid reasons why such a goal is not achievable. Using the Delphi 
Process, I will serve as coordinator of the effort. At the moment we 
are very near the beginning of Phase 1. 


Please review the preliminary plan outlined below and decide to 
become actively involved in what promises to be an intriguing effort to 
shed light on an important pedagogical and profession concern. 


ad hoc APA Sub-Committee on Testing Critical Thinking 


Draft Preliminary Plan 


Phase 1: Start the Inquiry 
The five objectives of this phase are: 


(a) Initiating the Delphi process. This is a method of achieving 
reasoned consensus among a group of experts with regards to a given 
problem or issue. The core of the strategy is to make inquiries, 
g.ther each expert's responses and their reasons, then summarize 
and share those with the group. After "hearing" what other experts 
think, people have the opportunity to refine their responses or 
defend those responses. The interactive rounds continue until 
reisoned consensus is achieved (or communications: break down). 


(b) Developing the "List of Experts" who will take part in this 
inquiry. Many of you were nominated by those who replied to my first 
letter, (2-11-88). That letter was sent to an original group of 
about twenty-five experts and interested persons suggested by 

the APA committee that conceived of this project and asked me to 
coordinate it. At any time if someone is no longer interested in 
continuing, just drop me a note. I plan to send out updated 

rosters of participants periodically. We are now up to fifty. 


wg 
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(c) Developing a bibliography on testing critical thinking. Work 
on this is progressing well. Many of you have sent me items to 
include, and I appreciate that. I will send you a copy of the 
bibliography later this semester. 


(d) Planning the subseq:ent phases in our process of responding to 
the general question of whether or not critical thinking (whatever 
CT is) can be validly and reliably tested at some educational 
level or levels. If the consensus is "Yes, at level X," then ve 
will focus on the question: How? If the consensus is "No, at leas 
not at level X," then we will focus on, "Why not?" : 


(e) Agreeing on basic assumptions.. 
In the 2-11-88 letter two assumptions were put to the group: 


Assumption #1: "Most of the members of our group [of experts 
participating in the Delphi*process] will come at the issue of 
testing critical thinking with the orientation of philosophers 
or logicians who teach at the post-secondary level, rather 
than as K-12 educators, psychologists, or personnel directors 
(all of whom also have legitimate theoretical and practical 
interests in assessing critical thinking). Given the 
interests of the American Philosophical Association, this [is 
an] acceptable [orientation] ." 


Assumption #2: "Critical thinking can be defined 
operationally to the extent that it can become a dependent 
Variable in a valid and reliable assessment tool." 


People were asked (1) if they agreed with the two assumptions 
as stated, or whether they would reject them or rephrase them 
somehow. Naturally, people were invited to explain why. They 
were also asked (2) to identify the educational level (K-post 
baccalaureate) to which our committee should give priority, 
and why they would recommend that level. 


Both questions have genereted controversy, as you will see in my 
next letter. If you haven't had the opportunity to respond to these 
questions, you will be invited to respond when the first round of the 
Delphi is reported back to you. Delphi is not about vote-counting, it 
aims at reaching agreement on the basis of reasons and common 
assumptions. In the Delphi method people are supposed to share their 
premises, not just their conclusions. 


As conceived at the moment, our work can be divided into 4 phases. 
These are not in stone! I welcome your suggestions, amendments, 
alternatives, etc. WE WILL USE THE DELPHI PROCESS TO AGREE ON OUR PLAN 
OF INQUIRY. Because the plan should be amended as a result of your input, 
only goals, not detailed objectives, have been developed so far for 
the next three proposed phases. 


ro 
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Phase 2: Define "Critical Thinking" 
The goal of this phase is to define "critical thinking" with 
sufficient clarity and precision to ask and answer the 
question of whether or not CT can be tested. Is CT 
fundamentally a set of skills, concepts, procedures, 
attributes, behaviors, outcomes, dispositions, aptitudes, or 
what? Even if we cannot reduce CT to an equivalent 
operational definition, how might we express what CT is with 
sufficient operational precisions to permit us to justifiably 
infer things about the relative CT abilities of students? 


Phase 3: Recommendations 
The goal of this phase is to communicate our findings about 
what CT is and whether there is an adequate way of 
characterizing CT operationally so as to permit its being 
tested as some educational level. Depending on our results in 
Phase 2, we will recommend either that programs aimed at 
testing CT be abandoned, or:that they be focused in certain 
ways. If this is the direction Phase 3 takes, then we will 
also try to come to consensus on recommendations regarding the 
relative importance of different kinds of CT sub-skills and 
possible strategies for accessing and measuring those sub- 
skills. 


Phase 4: Design and Validation of Model Testing Strategies 
Contingent on the results of earlier phases, the goal, if it 
were considered achievable in principle, would be to construct 
and evaluate different approaches to testing CT at some 
appropriate educational level or levels. We might find 
ourselves breaking into sub-committees to achieve this goal, 
although all work will have to be guided by the agreements 
reached in earlier phases and as well as by the special 
expertise of those who understand the intricacies of 
designing, piloting, norming and validating educational tests 
at specific educational levels. 


As I mentioned, you have been nominated as person who might be 
interested in this project and could make a strony contribution to the 
work of this ad hoc sub-committee. I hope you will agree to participate 
actively, because, as you must know, tie quality and utility of our 
effort is directly related to the involvement of concerned persons like 
yourself, 


Sincerely, 


Vz 


Pete Facione 
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Callfornia State University, Fullerton 
. Fullerton, California 92634 
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Department of Philosophy 
(714) 773-3611 
March 14, 1988 


Rovup 2 
eee 
Dear Testing CT Colleagues, 


Let's give Phase I, Round 2 a shot! 
CSS raa=azra 


Thanks for your responses to the first round of questions. Nineteen of the 
twenty-six or so who received the original 2-11-88 letter for Round 1 were able 
to respond. Round 2 invites everyone (which now includes just over 50 people) 
to review the results of Round 1 and comment on the agreements and controversies 
that are emerging. ° 

Round 1 focused on three issues; (i) The composition of our ad hoc 
committee in view of the interests of the Aaerican Philosophical Association, 

(2) the assumption that critical thinking can be Operationally defined, and (3) 
the educational level to which we should give priority. 


In regard to the composition of our committee, we are in decent shape, 
particularly since our group has been greatly expanded as per your 
recommendations. In regard to an operational definition of CT, we generally 
agree on the possibility but many would add various caveats. In regard to the 
educational level to which we should give priority, we have disagreements. 


The following pages cover each of the three questions in turn. You'll find 
restatements of the original questions and several representative quotes and 
Summaries of your comments. AFTER EACH SUMMARY, A SECOND ROUND QUESTION WILL BE 
PUT TO YOU. The new questions take the forn of stating a position and asking 
your opinion, now that you have had an Opportunity to consider what our 
colleagues have to say. In all, there are three new questions. IF YOU COULD 
GET YOUR RESPONSES TO ME WITHIN TEN DAYS OF RECEIVING THIS LETTER, THAT WOULD BE 





———_—~—=—————— i EN ee 0 ee 


GREAT! (I wish we all had electric mail, or unlimited phone budgets, but...) 


Several people noted that our task was huge, yet were Willing to give it a 
try. In contrast, one person wrote a major critique of the entire enterprise. 
This person argued that trying to test ST was a serious mistake. So that his 
Opinion is not lost in the shuffle, at the end of this package I have provided 
extensive quotations from his letter. If you find yourself in agreement with 
his views, then let me know and we will take up any “prior questions” we must. 
If you don't agree, then we will press on along the path we are charting for 
ourselves. : 


Thank you in advance for your partieipation. 
7s 
yours sintyrely 
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PHASE I, ROUND 2, FIRST ISSUE 
CURRENT STATUS: CLOSE TO CONSENSUS 


COMPOSITION OF THE AD HOC COMMITTEE: We were asked if we ayreed with the 
assumption that, although most of the membe.'s of our ad hoc committee would come 
at the issue of testing with the orientation of Philosophers or logicians who 
teach at the college level, this orientation still would be acceptable in terns 


_ of the interests of the American Philosophical Association. 


Almost everyone agreed, however some qualified their responses in terms of 
our collective professional interests and abilities, or in terms of educational 
level to which we should give priority. Here are representative responses: 


"I agree." "I have no problems with this assumption." 


"I don't see why this is a problem. First we're concerned about students 
acquiring the thinking skills required for college work,... Second, we're 
concerned that they learn the standards of good reasoning; I do not believe we 
need to know a lot about psychology to:achieve this purpose," 


"I agree, with reservations. We need to avoid tunnel vision. It is 
acceptable that most members be philosophers, but there should be a generous 
sprinkling of ‘outsiders' for the insights they will bring and to give our 
findings greater credibility outside the APA." i 


"We ave what we are! This is an appropriate place to begin. We are 
starting from what we know best and with what we can deal with most easily. 
This is not to suggest that we shall forever ignore other orientations, or that 
we really know that we can define all aspects of CT operationally." 


"I agree, this is acceptable; but it is unnecessarily narrow. Since so 
many of the tests are created by cognitive and educational psychologists, I 
think some of them should be included..." 


"It should not be too quickly assumed that those who teach at the post- 
secondary level are therefore knowledgeable and competent with regard to testing 
at the elementary school level." 


"I agree, but we should make a serious effort to inform ourselves of 
approaches to CT in pre-college and non-academic settings... Assuming our 
primary focus is everyday reasoning skills, we should not allow college CT 
instruction to be fundamentally different from pre-college CT instruction nor to 
become idiosyncratically colored by our own traditions. 


One person disagreed but did not give a reason. And one urged "Pete, get a 
proof-reader!" 


ROUND 2, QUESTION ONE: In view of the above comments, and in view of the 
additional names added as the result of your recommendations, can we agree that 
the ad hoc committee, as listed on the attachment, is sufficiently well- 
constituted for us to move on with our main task? As you can see, it still has 
its original orientation toward philosophers teaching at the college level, but 
it also includes several people from other relevant disciplines backgrounds, 
including psychology and education. 64 
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PHASE I, ROUND 2, ISSUE 2 
CURRENT STATUS: AGREEMENT SEEMS ACHIEVABLE 


OPERATIONALIZING CRITICAL THINKING: We were asked if we agreed with the 
assumption that CT can be defined operationally to the extent that it can become 
a dependent variable in a valid and reliable assessment tool. 


Here, too, most people agreed and were ready to get on with the work. Yet, 
some crucial ambiguities, concerns and caveats emerged. Here are some 
responses. 


"I expect we will argue about the details of any definition proposed, but I 
do not object to the assumption that we shall need some such instrument if we 
are to get any comparisons of interest." 


"This is a tautology because of the to the extent phrase. Perhaps this is 
how we should leave it..." 


"Sounds ok. ...I'm not a statistician, so I'm not quite sure what 
dependent variable means -- but if you’are asking whether CT can be tested, 
then, yes, I agree!" 


"I don't understand what this assumption is supposed to mean!" 


"I would agree only if we amended it to say at least some components of CT 
can be defined operationally to the extent that they can become dependent 
variables in a valid and reliable assessment tool... I do mot accept as analytic 
the proposition that CT can be defined operationally... I think some important 
aspects of CT, such as making judgment calls and weighting nuances may resist 
operational definition." 


"As the term operational definition is generally used by philosophers and 
education researchers, I do not think CT can be operationally defined... but I 
do think that part of the operational spirit can be employed in formulating 
reduction sentences (that do not reduce!)." 


"I agree, but there will probably have to be a variety of sub-definitions 
because CT is not one thing, but many. It somewhat resembles IQ in that." RE 


"There are several definitions of CT floating around... Some lend 
themselves more to operational definition than others.... If wa are to get 
anywhere, we will have to become clear in our own minds as to how CT is to be 
distinguished from other kinds of thinking..." 


"Is this a normative, definitional, conceptual, or planning assumption?" 


ROUND 2, QUESTION TWO: Without hanging ourselves up on the word “operations," 
can we agree that: (1) Even if CT cannot be reduced entirely to an equivalent 
set of operations for performances, behaviors, processes, outcomes, or skills,] 
(2) it is possible to conceptually analyze CT so as to describe a set of 
relevant and important CT operations, such that (3) using these descriptions, 
[competent] investigators could, on a consistent basis, gather sufficient 
evidence to draw conclusions, with high degrees of confidence, regarding the 
relative CT abilities of a group of people, (everything else being equal, of 
course]. 

62 


Appendix C: Delphi Research Letters, PAGE 59 


O 
z ERIC 





PHASE I, ROUND 2, QUESTION 3 
STATUS: CONTROVERSY 


PRIORITIES: We were asked which educational level (kindergarten through 
post-baccalaureate) should be the priority for our committee. And why so? 


Responses were split. Here are some representative examples: 


A person with considerable experience in the area of Cl testing wrote: "I 
think we should concentrate on high school at first. Since this is a 
subcommittee of the pre-college committee, all [levels] above that [are] ruled 
out. Furthermore, the younger the population, the more difficult the problems. 
Let's start with the easiest ones first -- and they are very difficult." 


By contrast, it was argued, "Priority should be given to the post-secondary 
level. One should examine end-products first, and then work backwards if 
needed. Find out first if the car doesn't run before attempting to determine 
Where the problem is. If a good test of CT revealed no CT deficiencies on the 
part of graduating seniors -~ (no doubt a counter-factual assumption) -- then I 
would think the APA might not wish to pursue the issue down in K-12." 


Noting that we are a sub-committee of the APA pre-college committee one 
person argued: "We must give priority to K-12: that is our mission." 


However, the person who will assume the chair of the APA pre-college 
committee for the next three years wrote: "It makes sense to start by playing 
from the APA membership's greatest experience and strength, which is surely 
college freshman level logic." 


Taking note of the interests of the APA, one person argued, "Since our ad 
hoc committee is convened within the structure of the APA, our focus should 
perhaps be primarily on the improvement of post-secondary education..." 
However, this person also suggested, "... that our assessment tool should be 
usable in secondary schools as well as at the post-secondary level..." 


Some people did not offer an opinion, but did note important distinctions. 
For example: "There are really two areas. One is the whole K-12 integration of 
thinking skills into the curriculum. The other is the single CT course, 
typically the approach followed in post-secondary education. The single college 
CT course offers exceptional opportunities for measuring gains in thinking 
skills, while the effort to incorporate thinking skills into the [K-12] 
curriculum may offer much greater potential for actually improving student 
skills." 


Another person, experienced in the pre-college arena, wrote: "It may be 
necessary to think of four tests, one for grades K-3 (one should not have high 
expectations for reliability at this level); one for grades 4-8 (the level at 
Which testing might have the maximum impact, even though the maximum impact for 
the teaching of CT might be at K-3); one for grades 9-12, and one for 13-16." 


Some Were tentative: ‘Perhaps we should give priority to CT at the college 
level, at least to start with, since the large majority of APA members teach at 
the college level. Latar we might wish to broaden our focus." 
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Some were direct, "College and university level." 


Others were focused, but concerned not to overlook anything important, 
"College level -- but someone ought to look at the high school level." 


One person declared for the college freshman level and argued against going 
any higher saying, "There are few if any thinking skills possessed by people 
beyond this level not also possessed by well-prepared college freshman. 

Graduate school and professional life chiefly consists in the ability to 
persistently apply these skills in more and more recondite subject matters. 


Some narrowed the range, but still left us with a choice: "I would say 
grades 9-12 and freshman/sophomore level in college." 


Another argued we should give priority to the introductory baccalaureate 
level saying, "First, it is the area where most philosophy departments have 
numerous classes actually being taught. Second, it is taught at a level which 
Will have the most connections in other areas and at other levels." 


ROUND 2, QUESTION THREE: To get started let's give priority to the college 
freshman/sophomore level. Do you agree? If not, is your disagreement based on 
pedagogical and theoretical concerns or on concerns relating to our charge as a 
Sub~committee of the APA committee on pre-college philosophy? 


Please try to get your responses to me within ten days. Thanks. 
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PHASE I, A CHALLENGE RAISED AGAINST THE ENTERPRISE 


One person wrote: "I would like to comment that teaching CT skills... isa 
far more significant matter than testing for them. Since CT is not an inherited 
trait, as is intelligence, the quality and extent of the CT is dependent on and 
proportional to the degree to which children have learned to, or have been 
taught to, think critically about their experiences and knowledge. 


"Our educational system ... is an abysmal failure. Most students are 
unable to recognize assumptions, not alone question or examine them. ... With 
rare exceptions, they show an almost total absence of recognition of even the 
simplest of logical/mathematical/linguistic/philosophical/scientific facts and 
concepts needed to be able to think critically. They have been so nurtured in a 
world of superficial "education" ... that to think of testing them on the basis 
of that "education" is an exercise in futility or at best an attempt to 
determine how inadequately they think critically as opposed to how much -- which 
if taken literally amounts to the same thing. 


"I am willing to contribute my expertise to the teaching of CT. I have 
been doing so for over thirtt years. But until I see considerably more evidence 
of students being able to think critically without such teaching, I see little 
point in testing them for the insignificant amount of critical acuity they may 
have acquired haphazardly. 


"... Teaching CT must precede and supersede testing for CT. Testing for CT 
cannot be considered to be an enterprise separate from teaching it. Testing for 
such skills and concepts presumes prior teaching of them. Psychologists and K- 
12 teachers as well as other educators show interest in CT. The problem 
remains, however, that most of them have only superficial, naive, and 
conflicting concepts, of what critical thinking entails. Even we philosophers 
can't agree on what it is... 


"From ay understanding of the term, testing for CT means testing based on 
what I teach CT is. ... Any tests that I would, and have designed, are 
predicated upon the version of CT I have taught. 


"My comments will undoubtedly reflect those which you will receive from my 
colleagues equally concerned with the problems facing our educational systen, 
particularly as they relate to the teaching of CT skills and concepts." 


REMINDER: If you believe there are issues (suggested by the above or otherwise) 
which our ad hoc sub-committee must address before we can move ahead, please let 
me know. On the other hand, the above challenge may represent a view which is 
not widely shared, or may raise questions which, in your view, do not fall 
Within the scope of our work or do not warrant our attention at this time. If 
that is the case, then, for the present, no response on this item is necessary. 


Thanks again for your participation. 


of À oo 
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Fullerton, California 92634 











Department of Philosophy 
(714) 773-3611 


April 14, 1988 


Dear Colleagues, 


Thanks for your help with the addresses. I also appreciate the 
notes, suggestions and other helpful comments many of you are 
sending. 


So far 17 responses to Round Tvo have been received. It would 
help if we had more, particulariy since it there may still he a 
split over Question Number 3, the one-about which level to pursue 
first. Drop me a note, with your reasons and opinions. Thanks. 


One of the responses to Round Two proposed an alternative to 
actually trying to come up with our own CT assessment instrument 
-- a goal some of us hope to achieve, but others of us are 
extremely skeptical about. This person suggested that we 
articulate the best list of CT skills we can, then let people go 
their own way with regard to building testing instruments. I'll 
expand on that idea when I summarize Round Two responses. I 
Mention it here because perhaps we all should be thinking ahead 
and trying to chart the most reasonable path for ourselves. 


Some of you have electronic mail. I don't, at least not yet. 

So, I'll be using snail mail and telephone to try to reach you 
for particular questions or clarifications. If you want to phone 
in your views on the questions in Round Two you can reach me at 
714-773-3742 (office) 08:30-10:00 MW or 09:30-11: 0 TTh (PDT). 

If those times ara inconvenient, call the department secretary at 
714-773-3611 and leave your phone number. I'll get back to you. 


I will be attending the First National Conference on Assessing 
Thinking in Baltimore on May 6 and 7. This conference is 
sponsored by the Maryland State Department of Education and the 
Association for Supervision and Curriculum Develonment. Although 
plans are to discuss all educational levels, the participating 
organizations, (over 35 professional associations, centers and 
government agencies) are concerned prinarily with K-12 education 
in some way or another. I'll report on what promises to be a 
most interesting gathering. 


within the week you should receive two ltéexs intrest. One is a 
partial bibliography CT with emphasis on testing CT, the other is 
an updated listing of some of the existing tests which purport to 
measure CT or closely related reasoning skills. 


Le 
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Department of Philosophy 
(714) 773-3611 
2 
(? OUND ~ 
May 4, 1988 


Dear Colleagues, 


I hope this letter finds you happy and well. It's time for 
Round 3 of our Delphi -- the round where we finally get to the 
heart of the matter: What is CT? Also, for those who are new 
to our effort, this letter includes a brief overview of who we 
are and what we are about. The last three pages summarize our 
Round 2 results. Please send Round 3 responses by June 25. 


As a result of nominations in Round 1 and Round 2, sixty 
persons, including some of the most eminent names in the field, 
are now invited to participate in this effort. 


By way of background, in January the American Philosophical 
Association Committee on Pre-College Philosophy asked me to chair 
an ad hoc sub-committee on testing critical thinking. Beginning 
with an initial group of APA nominees and asking them for 
additional recommendations, the "sub-committee" has grown to 
include people from a variety of academic disciplines and 
professional affiliations. Our unifying concern is in testing 
CT. However, we do not necessarily share the same 
conceptualization of what CT is nor do we necessarily agree on 
how it might best be tested. 


It is to resolve precisely these two things that we have 
undertaken the Delphi process. In Round 1 (Feb. 11, 1988) and 
Round 2 (Mar. 14, 1988) the focus was on establishing group 
membership and agreeing on preliminary working assumptions -- 

- such as the assumption that CT could be operationalized to the 
extent that valid and reliable assessments of important and 
relevant CT skills could be made. Starting with Round 3 we will 
focus on what those CT skills are and eventually we will decide 
On recommendations regarding testing, based on any Delphi 
consensus we achieve. To assist with the conceptual work that 
must be done, I developed and circulated two items, a list of 
existing CT tests anda CT-Testing Bibliography (Apr. 19, 1988). 


If you want copies of any of materials mentioned or if you 
wish to have a copy of the mailing/membership list, just drop me 
a line. 


Last week I presented a workshop on testing CT at Sacramento 
State. At that time Perry Weddle agreed to publish the CT- 
testing bibliography and the list of CT tests in a fall issue of 
CT News. So, please get any corrections, additions, or deletions 
to me as soon as you can. AND NOW ON TO ROUND THREE! 
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«*** ROUND THREE **** 
QUESTION: WHAT OPERATIONS ARE CENTRAL TO CT? 


Response requested by June 25. 





The sole task for Round 3 is for each of us to come up with 
that list of operations [or performances, behaviors, processes, 
outcomes or skills] which we understand to be at the core of the 
concept of Critical Thinking. 


Although many of us have published on this question, since 
there are potentially sixty in our group and since I have no 
assistants, it will be most helpful to me if you would take the 
time to distill your views and send a list of what you interpret 
the central CT operations to be, fell free to indicate which are 
the more general and which are the sub-operations. Naturally you 
are welcome to include justifications for the items on your list. 


In thinking about this, please keep in mind that in the two 
preliminary Delphi rounds we have narrowed our focus for now to 
CT "operations" understood as performances, behaviors, processes, 
outcomes or skills which could be tested validly and reliably at 
the college freshman/sophomore level. But keep in mind that we 
very likely will extend the question downward to K-12 later. 


In Round 4, which xï will initiate in Sept., you will be 
given combined lists and invited to comment on the wisdom of 
excluding, retaining, or amending the descriptions of specific 
items. If the results of Round 3 are clean enough, Round 4 will 
also invite you to begin rating items in terms of how more or 
less important, crucial, central, integral etc. they are to the 
concept of CT. 


If you do not intend to respond to Round 3, for whatever 
reasons, please drop me a line so that I can keep track of 
participation levels. 


Matt Lipman suggested that we might have -an excellent chance 
of working with the APA and the Assn. for Infornal Logic and CT 
te secure the use of the Wingspread Conference Center. Please 
let me know if you think it would be productive to get together 
in that setting. How might a conference be organized to most 
effectively use our valuable time? What kinds of issues, 
problems, tasks might we address? What kinds of solutions or 
desirable results might we achieve by meeting which couidn't be 
achieved (at all or as well) using the Delphi? 


Since Round 3 asks the "big" question, please take the time 


to respond. 
Fana prt 


Sincerely, 
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SUMMARY OF RESPONSES TO ROUND TWO 
Overall response rate: 23 of a possible 51. 


Question 1 of Round 2 asked if cur sub-committee was sufficiently well 

constituted to move on with our task. As you can infer, the consensus answer 
was "Yes." A small number of additional names from psychology and education 
were recommended and strong cases for adding ther were made. That is how we 
moved up to sixty members -- presuming the new nominees agree to join in. I'll 
send you an updated list soon. 


Question 2 of Round 2 asked if we agreed with this clain: 





"(1) Even if CT cannot be reduced entirely to an equivalent set of 
operations for performances, behaviors, processes, outcomes, or 
skills,] (2) it is possible to conceptually analyze CT so as to 
describe a set of relevant and important CT operations, such that (3) 
using these descriptions, [competent] investigators could, ona 
Consistent basis, gather sufficient evidence to draw conclusions, 
with high degrees of confidence, regarding the relative CT abilities 
of a group of people, {everything else being equal, of course]." 


There was sufficient consensus on this to move ahead. The majority of 
responses (18 were strongly to moderately positive, 2 were negative and 3 did 
not respond to this question.) 


However, to avoid misunderstanding, let us keep i.. mind that the above 
statement should not be interpreted to imply that construct validity can be 
determined strictly in an a priori manner (by simply coming up with our list). 
Nor should the words "relevant and important" be interpreted to imply that we 
can come up with an exhaustive list. Nor should the purpose in (3) be 
interpreted to mean that we have set our sights on actually writing a CT test, 
or, for that matter come to any agreement about what recommendations regarding 
testing we are likely to make. 


Most of the positive responses (13 of 18) were very short, "Yes," or 
“Agree," or "Yes, this is an acceptable working hvpothesis." Here are two of 
the longer positive responses I found interesting: 


"I agree with the statement offered. I am compelled to note thi’ the 
tasks described in the statement are going to be quite difficult to complete. 
There wili not be universal acceptance. There will be criticisms (legitimate 
and illegitimate) cf the results for a long time to come and many will .un a 
course Similar to criticisms of attempts to define and measure intelligence." 


"(1) Yas, even though ata =- thaueh T heliove vo con achiove a broadly 
satisfying reduction... (2) agreed here, (3) agreed here. Therefore, yes. And 
rather than get hung up on this question, I'd prefer to jump right in and see 
whether we can do it. If we can, terrific! If we can't, well, then the 
doubting T's will have a field day; but I'm prepared to take that risk..." 


One of the negative responses was expressed this way: "I'm.sorry, but I 
can't help but get hung up on ‘operations’ -- the term so psychologizes ard 
jargonizes... the question. Why not go for 'principles'?..." 


The strongest negavive was registered by a person who said, "I fear this 
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proposal to define CT operationally may have the effect of ruling out, by 
definition, one major position on CT, a position with which many in the CT 
field may be in sympathy, at least to some degree." This person was concerned 
that defining CT as a list of skills would focused on weak-sense-CT and missed 
strong~sense~CT which relates to a person's character -- "being self-critical, 
seking to overcome blind spots, biases, prejudices..., [being] critics of 
one's society,... (seeking) what is of value in another's position..." The 
person asked, "Can these character traits be defined operationally?" 


In contrast, note this response: "Yes, [I agree with the statement] 
particularly if we confine ourselves to abilities and ignore dispositions..." 


Even though agreeing that "A subset of the processes that constitute CT 
can be assessed using the multiple-choice format that I assume is being 
sought," one person expressed serious concern saying, "I fear the creation of 
an instrument promising more than it can deliver -- an instrument touted not 
for what it is, an assessment device measuring certain important, but 
rudimentary, CT activities, but rather as a valid and reliable assessment tool 
for CT." He maintained that "As is so commonly true when discussing 
assessment, the instrument and its characteristics would then dominate the 
social construction of what is being assessed, in this case CT. The initiation 
of CT activiti¢s, generation of appropriate CT strategies, and defense of a 
tentative reasoned judgment are not susceptible to the type of assessment 
legislators are willing to finance or faculty are willing to undertake. The 
extensive writing or oral argument required to demonstrate CT, as I understand 
it, are not practical inclusions in an assessment instrument." 


Four who agreed with the approach mentioned important factors which relate 
to construct validity. One mentioned the role of background knowledge in CT, 
another the role of divergent assumptions, another the relationship of CT 
skills to reading skills, and a fourth spoke to the need to validate any list 
of CT operations we might agree. There is much in the research literature 
about these problems. Steve Norris, in particular, has been working on 
strategies to respond to precisely these kinds of problems. 


Although I mentioned all the negatives, the positive responses were far 
more numerous than the negatives. A consensus to move ahead exists. But we 
wust not forget the warnings and concerns of our colleagues. A great deal 
depends on what we come up with when we actually sit down to answer the 
question for Round 3, since both the positives and the negatives were based on 
our ideas about what CT is. 





Rou. , 2 question 3 asked if people would be willing to agree my proposal that 
to get started by giving priority to the college freshman/sophomore level. 


The responses ran: 14-yes, 3-no, 2-both, 3-abstain, and one that I could 
not figure out. Since the question was about priority and was not intended to 
exclude working at the K-12 level, which is, after all, what the APA Pre- 
college Committee is charged with doing, I believe we have sufficient consensus 
to focus initially on the lower division post-secondary level. 


Here are some "Yes, give priority to the frosh/soph level" comments: 
“Most if not all of the CT we teach is directed at this level" 
"I doubt philosophers should take the lead...when it comes to K-12." 
“I'm still unpersuaded to reverse my formerlv expressed views -- 


Appendix C: Delphi Research Letters, PAGE 67 


u 7 0 





JENS 


intro. undergraduate level first, until we get square there, 
where we live." 

"I feel rather strongly that we should begin on the post-secondary 
level. I agree with those who say that that is where the 
strength of our membership lies and that that is where the vast 
majority of our members teach." 

"Yes. That's where most of the pedagogical action is; that's where 
the students we're interested in testing and have relatively 
easy access to are to be found. Later, if we succeed at all 
here, we can extrapolate to other levels." 

"I agree in the light of what was said in Round 1." 

"Why not? It's what we know best, and we can always move on to other 
levels later." 


‘On the negative side: 

"(Your] recommendation seems arbitrary and not consistent with the 
fact that this is a pre-college committee." 

"No, based on the name of the committee." 

"I would prefer to begin at the K-3 level. My position and 
opposition is based upon pedagogical and theoretical concerns 
which I assign a higher priority to than to political concerns 
related to the officers and membership of the APA.... I am 
willing to accept that the sub-committee begin with the college 
level but the project will have to be extended downward then..." 


Two people expressed the concern that this was a difficult question to 
answer until one knew the purposes for testing. [A point well taken.] 


Another suggested that our goal should be to make "contributions to the 
criteria for a college test...(but that] individuals should be encouraged to 
make up their own [assessment instruments] and try them out, obtain 
correlations with other tests and with outcomes, and then subsequently compare 
notes with one another as to what worked and what didn't.” 


Thanks to all who responded to Round 2. Your 
letters were most interesting and thoughtful. 
Don't forget ~- Round 3 by June 25! 


7a 
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Department of Philosophy 
(714) 773-3611 


May 18, 1988 


Dear Colleagues, 

I hope this letter finds you well, happy and looking forward 
to a restful as well az productive summer. In my last letter I 
mentioned that I would be attending the "First National 
Conference on Assessing Thinking" in Baltimore and that I would 
be sending along a report. The conference was very rewarding and 
the promised report is attached. 

I've also attached the current list of persons invited to 
respond to Round 3 in our Delphi process. We will have to freeze 
the list at this point. According to what I've read about the 
Delphi method, once the central debate is joined, it can be 
disruptive to try to add people ahi have not the benefit of 
earlier rounds in the dialogue. With the circulation of the 
Round 3 question we have reached that point. 

Many thanks to those who have already sent there responses 
to Round 3. Don't panic if you haven't yet, though. The target 
date is June 25. 

Have a good summer. I look forward to hearing from you. 

Sincerely, 
= 
Lee | 
Pete Facione 
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California State University, Fullerton 
Fullerton, Califomia 92634 
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Department of Philosophy 
(714) 773-3611 


June 28, 1988 


Dear Colleagues, 


Thanks for the large number of interesting responses to the 
Round 3 of our critical thinking Delphi process, Judging from 
the length and sophistication of what many of you sent, it should 
take me a fair amount of time to Organize and synthesize the 
material and then to frame fruitful Questions for our next round. 

I expect to be working on this for several weeks, so if for 
some reason you haven't had the opportunity to respond to Round 
J, please know that your ideas are more than welcome. 

Have a good summer and thanks again for so much high quality 


Participation -- it’s very encouraging. 


Sincerely, 


Jo 


Pete Facione 
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Department of Philosophy 
(714) 773-3611 


Sept. 1, 1988 


Dear Delphi Colleagues, 

l'm working through the 25 responses to Round 3, which asked for your 
List of core CT skills. fhe Scope, quality and care evident in your responses 
is most impressive. Right now l'm analyzing, synthesizing, and organizing the 
ideas so that I can play them back for your reaction and rectification, 

Expect the Round 4 letter in about two weeks. 

A Delphi operational rule I've adopted is ta drop persons off the Delphi 
mailing list if they have made no contact whatsoever after receiving 
invitations to respond to at least three Delphi rounds. It’s fair to say a 
Persan’s sustained silence signals he or she wishes not to be included. 

Included here is something Phil Pecorino Shared. It lists (a) Critical 
Reasoning and Informal Logic Teats, (b) Related Texts, (c) Logic Texts, (d) 
Newsletters and Journals related to CT, and (e) CT Centers and Organizations. 


Also, you might also want to write to James Bell, Howard Community College, 





Columbia, Maryland, 21044 and ask for a copy of his 132 page Guide to CT for 
Maryland Social Scientists. It includes a wealth of atari on how CT is 
seen from different disciplinary backgrounds and it lists a number of ideas 
and resources, 
Thanks again for so much high quality participation in Round 3 -- it’s 
very encouraging. You'll be hearing from me soon. 
Sincer elu 


Ees 


Pete Facione 
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Department of Philosophy Sept. 23, 1988 
ROUND 4 


Dear Delphi Colleagues, 


Round 4 seeks verification of a list of CT skills and sub-skills. 
Please accept, reject, amend, and comment on the group's responses to 
round 3. Remember, the goal in this phase of our project is to arrive 
at an accord regarding the skills we understand to be central to CT. 


Your responses to round 3 yielded 200+ pages. Some sent previous 
publications, some sent lists and commentaries, some wrote new pieces 
of clearly publishable quality. You tended to approach the question 
of identifying core CT skills and sub-skills four ways: (a) by appeal 
to your own experience and understanding, (b) by citation and comment 
on what other CT authorities (including others in our Delphi list) 
have written or said, (c) by describing the key characteristics of 
persons who have internalize CT, and (d) by consideration of what 
should reasonably be taught or included in a CT curriculum. In 
addition to differences of opinion, there were variations in 
disciplinary orientation, vocabulary, and emphasis. There were also 
differences in the specificity, depth, and scope of responses. Some 
were extremely general, others very specific. 


Distilling your opinions, positions, views, ideas, lists, 
descriptions, explanations, examples, counter-examples, caveats, . 
credos, and course outlines was one of the most intellectually 
interesting and stimulating experiences I’ve ever had the pleasure of 
attempting. Although I’ve been = .iching and researching CT for two 
decades, I noticed that my Own views on the range and character of CT 
expanded greatly as a result what you contributed in round 3. 


ln naming and describing CT skills for Round 4, I intend to rely 
on standard English usage and to avoid technical or discipline- 
specific vocabulary. Your responses emphasized generic skills. 
Distilling your responses, I name and describe six generic CT skills 
and give two or three sub-skills under each. Clearly additions or 
deletions might be needed. To avoid prejudicing your responses at 
this crucial time, I do not indicate the numbers of persons who may 
have agreed on any given point. Areas of agreement and controversy 
will come out in round 4. In this round you are invited to make a 
number of kinds of responses to a variety of questions. However, 
because organizing the material and framing the issues was such a 
delicate and complex task, I urge you to read the whole package and 
get an overview of the terrain prior to starting to reply. 


Thanks again for the high quality responses to Round 3. To 
insure we are talking about the same things when we refer to CT 
skills, we'll need maximal participation in round 4. If you could 
consider this material and reply in 15-20 working days that would be 
wonderful. If you need more time, or want to discuss any aspect of 
this project, call me at CSUF (714) 773-3742, [office] or 372-3611 
[dept.], or 993-1356 [home]. 


F pe 
Sincerely, -N . S 
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There are five parts to Round 4; 
(1) The listing of CT skills and sub-skills. 
(2) A model diagramming the flow of CT skill input and output. 
(3) A list of things some of you said CT is not. 
(4) A description of what is meant by "Skill". 
(3) A list of caveats and comments you shared. 


Round 4: Part 1, CT skills 





Comment: We've all seen examples, like the duck-rabbit, of how comprehension 
combines imposing an order on reality as well as discovering an order inherent 
in reality. Applying that lesson to the problem at hand, there are many ways 
CT skills and sub-skills could be organized. Even among those of you who 
essentially agree, the variations in your responses to round 3 illustrate 
this. After considering a large number of ways of organizing things, I've 
decided on the configuration you will find below. True, the skills and sub- 
skills could have been named using other labels or grouped in other ways. To 
this add that some sub-skills may come into play in more than one ‘general 
ar@a, some can operate on the products of others, some presuppose others, some 
are almost always employed while others may not come into play except under 
special circumstances, and you have an even more conceptually complex 
Situation. Now add that some of us might. exclude one of the more generic 
groupings, to say nothing of how we might want to add, subtract, amend or 
rearrange the sub-skills, and the complexity of our task takes on greater 
magnitude. There were other ways to "see" the Round 3 data. So, beside 
asking yourself if each skill and sub-skill] belongs in the iist, and if 
anything central is missing, another question to ask is, like the duck-rabbit, 


can you see CT this way? 


Instructions: Belon you will find aelist naaing and describing six core CT 
skills and subeskills, Read and consider the entire list. Then, 

(1) Considering each skill and sub-skill, marl. each "Yes!" if you verify 
it to be a core CT skill or sub-skill and "No" if you would arque to exclude 


it. State your reason for excluding any marked "No". 
(2) Wake needed substantive amendments to the descriptions of. any sub- 

shill, and explain why the change is needed. Add any missing sub-skill, name 

and describe it, locate 1t within one of the six CT skills. Considering the 


subeskills within each of the six CT skills, indicate which sub-skilis, if 

















anys, Should be moved to some other skill. 
(3) Amend the list of six CT kills, lf you delete a skill, indicate 

















d 
CT. how it differs from any of the six skills on the current list, and why 1 
merits being listed at the level of a shill rather than a sub-skill. 

(4) Consider the NOTE regarding the role and composition of a human's 
kKnunledge bate, verify, amend, comment on any aspect of this. 

(5) Make any needed editorial changes. 

(6) Comment, if you wish, on the entire list of skills, its organization, 
utility, what have you. 








l) urge vou to read the entire list of 
skills and sub-skills before beginning to 
respond to any specific item. Thanks! 
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PROPOSED: Core Critical Thinking Skills 


TAXONOMY 
la Interpreting ~- Observing, Decoding, Clarifying. 
ce Inferring -- Querying, Conjecturing, Drawing Conclusions. 
we Analyzing -= Locating Arguments, Farsing Arguments. 
4. Evaluating -- Verifying Claims, Assessing Logical Strength. 
we Expressing —- Stating Results, Describing Procedures. 
ó. bKonmitoring -~ Regulating, Reviewing, Correcting. 


DESCRIPTIONS OF SKILLS AND SUB-SKILLS 
YES/NO 


1. INTERPRETING: To comprehend the significance of a wide variety of 
experiences, situations, judgments, beliefs, rules, procedures and 
criteria. 

1.1 OBSERVING: To detect, attend and correctly perceive 
experiential input with particular focus on input that conveys or 
is intended to convey data, information, or inferential 
relationships. 

1.2 DECODING: To detect, attend to and correctly perceive 
the informational content, rules, procedures, criteria, and 
inferential relationships expressed in various convention-based 
communication systems, such as language, social behaviors, 
drawings, numbers, signs and symbols. 

1.3 CLARIFYING: To make explicit, through stipulation or 
description, the contextual, conventional and/or intended 
meanings of words, ideas, concepts, statements, behaviors, 
drawings, numbers, signs or symbols; to remove confusing. 
vagueness and ambiguity; to facilitate communication. 


2: INFERRING: To secure elezents needed to sake inferences and ty 
detersine the inferential relationships between or fioning troa 
statements, descriptions or representations. 

2.1 QUERYING: At any point in the CT process, to recognize 
the need for evidence or information of some kind, and to 
formulate and execute a strategy for seeking and gathering that 
evidence or information. 

2.2 CONJECTURING: To formulate alternatives, to develop 
hypotheses, to postulate suppositinns. 

2.3 DRAWING CONCLUSIONS: Given a set of statements, 
descriptions or representations, to educe their inferential 
relationships and to educe the consequences which they 


support, warrant, imply or entail. 
FY pow 


we 
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* *. š 
NOTE: As many of you argued, all CT skills, but particularly sub- 
skills like 2.1, 2.2 and 2.3 presume a knowledge-base, A human's 
knowledge-base is composed of at least these things: 
(a) a world view which includes one's understandings of 
* what is real (a metaphysics), , 
* how knowledge is gained and refined (an epistemology), 
# what is important or valuable (a value theory); 
(b).a data base. including one's opinions, beliefs, 
experiences, etc. as filtered through the world view; 
(c) an inference engine which includes 
* general rules for drawing logical inferences (a logic), 
* sets of procedures and criteria appropriate for making 
reasonable judgments within specific areas of human 
thought and inquiry (discipline-specific rules.) 














Executing sub-skills 2.43, 2.2, and 2.3 as well as 4.1 and 
4,2,'more effectively can be achieved by learning how to think 
logically, by expanding one's repertoire of sets of procedures and 
Criteria used in different areas of human thought and inquiry, 
and increasing cne‘s base of relevant data. 

An implication of this analysis of CT skills and sub-skills 
is that they transcend specific disciplines, but executing them 
demands background knowledge, some of which is specific to how one 
goes about making reasonable judgments in different realms. 

Becoming adept at CT involves learning CT skills and 
learning to use those CT skills more effectively in different 
contexts -- hence the importance of a liberal education to go 
along with one's CT ability. 

* * + 


S. ANALYZING: To identity the inferential relationships between 
statements, descriptions or representations which express experiences, 
situations, judgments, beliefs, or opinions. 

3.1 LOCATING ARGUNENTS: Given a set of statements, 
descriptions or representations, to determine whether it does 
express or was intended to express a reason or reasons in support of 
some claim, opinion or point of view, 

3.2 PARSING ARGUMENTS: Given a the expression of a reason 
Or reasons in support of some claim, opinion or point of view, to 
identify: (a) the intended conclusion, (b} the premises and 
reasons advanced in support of that conclusion, (c) additional 
unexpressed elements of that reasoning, such as intermediary 
conclusions, unstated assumptions, and (d) for exclusion, any 
items contained in body of expressions being parsed which are not 
intend to be taken as crucial to the reasoning being expressed. 


4. EVALUATING: To assess the credibility of statements, descriptions or 
representations; and to assess the strength of the expressed inferential 
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relationships betneen such statements, descriptions or representations. 
4.1 VERIFYING CLAINS: To assess the degree of confidence 
to place in a given statement, description or representation, 
4.2 ASSESSING LOGICAL STRENGTH: To determine the nature 
and quality of expressed inferential relationships; to judge 
whether the assumed truth of the premises of a given argument 
justify one’s accepting as true, or very probably true, the 
expressed conclusion of that argument, 


J: EXPRESSING: Ta state, describe or represent to one's self ar athers 
the results of one’s CT activities and the way one nent about producing 
those results, 

5.1 STATING RESULTS: To produce accurate statements, 
descriptions or representations of the results of one’s CTI 
activities so as to analyze, evaluate, infer from, monitor or 
remember those results, or so as to communicate them effectively to 
others. 

5.2 DESCRIBING PROCEDURES: To produce accurate statements, 
descriptions or representations of how one applied and executed 
any CT skill or sub-skill so as to evaluate or monitor one's 
proficiency, or so as to communicate to others about how one went 
perfcrming a given CT skill or sub-skill. 


ô: MONITORING: To regulate all aspects of one’s onn CT activities, the 
elenents used In those activities, and the results produced by those 
activities, particularly by applying the skills ef analyzing, and 
evaluating to one’s own inferring with a view toward confirming, 
validating and/or correcting the results, 

é. 1 REGULATING: To sequence one's executinn of CT skills 
and sub-skills. 

6.2 REVIEWING: To examine one's own CT activities and verity 
both the results produced and the correct application and execution 
of each CT ski id and sub-skill involved, 

6.3 CORRECTING: Where errors are found in one's own CT 
activities, to covrect those errors and remedy their causes. 
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Round 4: Fart 2, An Input/Output Model of CT Skills 








Comment: There's no denying that the model I’ve come up is heavily 
influenced, for good or ill, by my reseerch in computer science and 
artificial intelligence. The arrows imdicate the direction af the 
flow of input and output as it circulates between and is operated upon 
by the six CT skills, here conceived of as functions. 


In this model, expressing receives output in the form of 
information from the knowledge base, the results of other CT 
funckions, and lists of the steps performed during other CT functions. 
kupressing then formats that output for use by the next CT funckion, 
for storage in the knowledge-base, or for transmission out of the Ul 
cycle. 

Within the CT cycle, monitoring receives material from expressing 
and, as the regulating and self-correcting function, determines where 
it should go next. .It can route things into or aut of any of the five 
other CT skills or the knowledge-base. For example, monitoring can 
loop material back through any skill. Thus, it can send the resulte 
of one's own inferrine 
allowing expressing to transmit them to others. Gr, it can route 
information from the knowledae base to evaluating to help it verify a 
claim, or to inferring to help it draw a conclusion using criterie 
specific to a given discipline. 


The four CV skills many of us spend so much time helping estudents 


eame: 


They operate on this material. And then they send the results to the 
expressing function to be formatted for delivery elsewhere. These 
Skills also output a record of the steps they performed in coming to 
those results. This record is crucial if the monitoring function is 
to work correctly, since it must check not only what was achieved but 
how it was achieved. 


The khewledge base is a storehouse from which interpreting, 
analyzing, evaluating and inferring draw resources. lt stores the 
output at any CT funckion, when directed to da so by the monitoring 
function. The knowledge base also filters raw external input and, 
thus., influences the interpreting of what we are observing, decoding 
or Glaritying. l 


Instructions: Consider 


1. Does it make sense to you? 
3 it accurate? 

me ds Lh useful ?. 

4. Since many people find pictorial models helptul in 
understandina cample: relationships, can you suaqgest how this ane 
might be improved or, if you recommend discarding it, can you propose 
an alternative? 
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Round 4: Fart Z, What CT is Not 





Comment: Many of you distinguished CT skills from other closely 
related things. Below is a list of what various persons said was 
not CT. dust because something is an this list does not mean a 
person goes nok use Cr in doing the thing nor thag a person might 
not do the thing befure, after or during CT. 


SOMOS: SOLS MACS ES Cen Ge ammen scenes senna mEenonys osm sunon 


memnon mwponump anen ey ee ameen ormsa. a 


CRITICAL THINKING IS NOT 


YES/NO 


l. Sensing, (Seeing, Touching, Hearing, etc.) 

2. Reading, Listening, 

J. Speaking or writing, 

4. Motivating, persuading, selling, 

5. interrogating, cross-examining, petitioning, 

9. Physically investigating the world around, 

7. Trouble-shooting, problem-selving, puzzle soiving, 

8. Decision-making, selecting, choosing, exercising§ one’s will, 
9. Planning, defining goals and objectives, 

10. Finding or ascribing a meaning to art, life,.or events, 
11. Defending an opinion or belief, arguing a cage, 

tz. Managing, administrating, cr governing persons or things, 
13. Fhilosaphizing, 

14. Conducting research within any particular discipline, 

1S. Experiencing, feeling, emoting, or empathizing, 


16. Communicating using language. 
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Round 4; Part 4, The Concept of a Skil 
Comment: It became clear from your responses that it would be useful to have 
an understanding about what a skill is. Although there may be very little 
disagreement about this, some of you mentioned subtleties others may or may 
not accept. Based on your contributions I’ve written a little narrative, 


Instructions: Revise, edit and complete the following narrative: 


ee Ee p 2 


A skill is the ability to do something well. Having a skill includes kKnoning 
Nhat to du, when to do it, and hon to do it. That is, being skilled at 
Something involves knoning a set of procedures, judging nhen to apply those 
procedures, and being proficient or adept at executing those procedures, 
Skills can 1 aught in a variety of mays including, but not linited to, 
coaching, demonstrating, and training. Part of the teaching involves aaking 
the procedures explicit and showing when and how they are enployed. Skills 
can be learned through a coabination of observation, guided practice, drill, 
and self-corracticn, Persons can be judged as being pore or less proficient 
in a given skili. The first way of assessing 1s to observe the skill as it as 
being perforaed. A second may ig to coapare the outcoses (if any) that result 
froo executing a given skill against soge set of criteria. A third way Is ta 
query persons ang receive their descriptions of the procedures and judgments 
they are using as they perfore that skill, would use if they were to perfora 
that skill, or did use when they perforeed that skill. Sirce N2 cannot 
directly observe the perfornance of sental skills the may me can physical. 
Skills, only the second and third ways are available toa those nho would assess 
CT. The secaend nay rs tis not) superior to the third DECAUSCesvccnesncey 


provided Sin: eae a er a ae 


84 
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Round 4: Part S, Caveats and Cautions 











Comment A number of you sent conuentss, Some were intended as caveats or 
cautions, other as encouragement. I appreciate them all. 1 thought some 
might be good to share: In many cases to save space | have paraphrased. You 
might agree or disagree with what your colleagues have said. Or, reading this 
‘list might prompt you to pass along a contrary view. Let's find out. 


Instructions Consider these comments. Nake any remarks you wish, whether in 


agreement or disagreement, Use these remarks as.spring boards for your own 
reflections. 








AGREE/DISAGREE 


1. Keep in mind that our goal in defining CT is to do 
some assessment. But you don't just start testing people. 
Assessment needs focus and purpose. 

2. I agree with working at the college level only because 
l want to have an idea about what we should be doing with kids 
in K-12. I think we can use what we say about CT for college 
frosh/Soph to guide curriculum development in Kel. CT. Of 
course, we will have to adjust reading levels, background 
knowledge expectations, and lots of other things. . 

S. Skills are not the Same as operations. Resist 
behaviorism! Behaviors give evidence that a person has a 
skill or ability, but a skill is not a set of behaviors. 

4. Don't trap yourself into using the jargen or 
vocabulary of any one discipline (especially philosophy) when 
you describe CT. 

Se Although no word will escape criticism, don’t use 
"deduction" or "induction." Avoid semantic spats. 

6. If CT is a set of attitudes as well as a set of skills 
that poses no problem for assessment because we can just 
develop ways to assess the CT attitudes, too. 

7. CT is thinking skills. Saying CT is a set of 
attitudes may be a way of describing what people who are good 
at CT are like, but it is not a way of describing critical 
tF king itself. 

8. Even if we agree on whst CT is, we still have to face 
the problem that any student might get the right answer on a 

` CT test but for the wrong reason, or might get a problem wrong 
but have done a good job of CT. 

9. When assessing CT we should not duplicate efforts with 
areas already well tested by existing instruments, such as 
covered by reading or intelligence tests. 

10. You don't have to test every ingle CT sub-skill to 
decide that è person is good at CT. 

1i. I looked at that list of exper.s and you have all the 
big names I can think of, but you can’t possibly expect those 
people to agree. If they did agree, even on what CT is, that 
would really be something. Good luck! 


FEEL FREE TO DISCUSS THIS MATERIAL WITH COLLEAGUES, IF YOU WISH, 


PLEASE DCN'T REPRESENT ANY OF THIS LETTER AS THE OPINION OF OUR DELPHI GROUF, 
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California State University, Fullerton 
Fullerton, California 92634 





Department of Philosophy 
(714) 778-3611 


Dear Delphi Colleagues, 


During this Thanksgiving season and I want to express my gratitude to 
you for your generous participation in this research during 1988. The many 
responses to the very long and difiicult ROUND 4 have been most gratifying. 


There won't be any more "ugly-long" rounds like that, I promise. 


Let me pass along this quick review of what we accomplished in 1988. 
First, working under the auspices of the Ameri~an Philosophical Association 
Committee on Pre-College Instruction in Philosophy, we built the Delphi 
list of experts. By your recommendations, during Rounds 1 and 2 (Feb. 11 & 
Mar. 14) we expanded the original APA list of about ten names to a working 
group which numbers around 45 active participants and which, I am proud to 
say, includes many of the most important people in CT research today. 


During Round 1 and 2 we agreed that whatever CT is, we would be 


_ possible for us to make the concept operational to the extent that 


important parts of CT could be assessed validly and reliably. We also 
agreed to begin by identifying the core elements of CT expected at the 
Frosh/Soph. general education college level. We agreed to use this college 
level theoretical-construct of CT to guide what is said about CT the K-12 
levels. Round 4 (Sept. 23) sought to verify the concept of CT which 
emerged from Round 3 (May 4). A quick look at the results of Round 4 is 
most encouraging! 


Along the way we shared journal articles, lists of existing CT tests, 
CT bibliographies and other items of mutual interest. ‘fhe Delphi, however 
is not a substitute for the fine work being done by jecurnals, newsletters 
and the many centers for CT that have emerged in recently. 


While I work on analyzing the results of Round 4, I invite you to 
consider where do we go from here. Last spring I outlined a four phase 
Delphi project. When we achieve consensus on the core list of CT skills 
expected at the lower division college level, we will have completed the 
two of the four phases. Originally phases 3 and 4 were described this way: 


Phase 3: Recommendations 

The goal of this phase is to communicate our findings about what 
CT is and whether there is an adequate way of characterizing CT 
operationally so as to permit its being tested at some 
educational level. Depending on our results in Phase 2, we will 
recommend either that programs aimed at testing CT be abandoned, 
or that they be focused in certain ways. If this is the 
direction Phase 3 takes, then we will also try to come to 
consensus on recommendations regarding the relative importance of 
different kinds of CT sub-skills and possible strategies for 
accessing and measuring those sub-skills. 


Phase 4: Design and Validation of Model Testing Strategies 
Contingent on the results of earlier phases, the goal, if it were 
considered achievable in principle, would be to construct and 
evaluate different approaches to testing CT at some appropriate 
educational level or levels.. We might find ourselves breaking 
ino ‘sub-committees to achieve this goal, although all work will 86 











have to be guided by the agreements reached in earlier phases and 
as well as by the special expertise of those who understand the 
intricacies of designing, piloting, norming and validating 
educational tests at ‘specific educational levels. 


To date I've done little regarding preparing to communicate our 
findings. Two CT newsletters interested in publishing something regarding 
our results have contacted me. Also, the Pre-College Committee scheduled a 
session at the March 1989 APA meetings at UC Berkeley. At that session 
I'll be outlining our Delphi process and what we have agreed on by tnat 
time. More suggestions are most welcome. 


For many reasons I am extremely skeptical about actually developing a 
good CT test using a Delphi process. Once we declare consensus regarding 
the theoretical construct of CT for the general education (lower division) 
college level our choices include at least these three, and maybe more. 


(1) We could move on to consider questions like these: Given what we 
understand CT to be at the college level, what does CT mean a` different 
grade levels in K-12? What is the relative importance of the skills or 
sub-skills in our college-level CT construct in terms of testing, say 
junior high school students? How might one write a question which assesses 
a given sub~skill in, say 5th graders? 


(2) Having declared consensus on a conceptualization of CT for use at 
the college level, we might recommend that test makers at all educational 
levels be guided by our conceptualization. But we, ourselves, might decide 
to leave the matter of writing specific tests for specific age groups to 
others, better qualified than ourselves for developing and validating such 
instruments. Those of us interested in specific grade levels could be put 
in contact with one another. 


(3) When we reach consensus on the CT concept as it applies at the 
college level, we might recommend examples of how questions framed 
to address these skills and sub-skills in college students or K-12 
students. We could share these example questions and evaluate them. Those 
which we think a priori might be good to assess certain skills or sub- 
skills, could be included in recommenéations we make regarding CT 
assessment. These questions would not be a CT assessment tool. At best we 
might think of them as models of how to cenceive of questions that might be 
included in a CT assessment. 


Note: Even with questions which a priori seem to address the proper 
concept of CT and avoid other difficulties (like relying on special 
background knowledge or esoteric vocabulary), there is still the problem of 
a posteriori verification. Steve Norris has done important work on how to 
overcome the "construct-validation" problem, namely determining, for any 
given test item, if students get right answers because of good CT skills 
and wrong answers because of inferior CT skills. 


Give the issues of what do to next some thought. I would welcome 
hearing from you on this. Have a joyous holiday season and thanks again 
for contributing so generously during phases 1 and 2 of this project. 
Sincerely, 


S a a 
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California State University, Fullerton 
Fullerton, California 92634 


Department of Philosophy 
(714) 773-3611 





Feb. 7, 1989 
Dear Heroic Delphi Colleagues, 


34 people responded to Round 4. A great return, our biggest so far ~-~ 
: both in numbers of persons and in numbers of pages! 1} will tally, analyze, 
distil) and shere what people said on each chur: of Round 4. Ihat way you 
will have the benefit of an overview of what others in the Delphi are 
thinking. Hut, intending to spare all of us any more horrendously 
burdeusone rounds, when it comes to formulating the next set of questions, 
1'11 keep them as short and focused as possible. 


l particularly want to thank the many people who sent me detailed, 
thoughtful, (and even fuutnoted) responses. Several ran near ten pages 
Single spaced. Not intending to diminish the value of the brief "Yes/No" 
responses aS we approach consensus on crucial points, I must acknowledge 
and convey my appreciation for the many extra hours of work several of you 
are putting in. Also, l've learned a great deal from your sensitive and 
sensible comments. 


Yes, we are approaching consensus. My first two readings of the input 
on Round 4 is that we have a great deal of accord on the list of CT skills, 
although there will be scores of adjustments and amendments to be made, 
Also, your comments on what a skill is and how a cognitive still can be 
assessed suggest we are close there too. But J'1) be summarizing all that 
and more for you very soon. 


In the next few weeks I‘)1] be sending you a few quick short rounds. 
iy plan is to focus each brief letter on one discrete aspect of our work. 
We should try to complete several mini-rounds this spring. That way 1°1] 
be able to rough out a first draft of our report to the APA Fre-College 
Philosophy Committee during the summer. se thinking about recommendations. 


I've enclosed a list of the people who are participating in our Delphi 
research project. The "Ki" "R2" "RS" and "K4" symbols indicate the rounds 
to which the person has cuntributed. The "i" symbol means the person 
communicated an interest to be involved, but has not responded yet to any 
rounds. Currently there are 56 names on the list uf people being invited 
to respond. A few have never responded in any way. So, when it comes to 
making our final report, | expect there to be around 50 of us in the group. 


T')) be in touch with a summary of Round 4 and some mini-rounds 5, 6 
etc. very Soon. Thanks again for all your work. 


Sincerely 
oa 
— 
r aT 
AAEE 
Feter A. Fatione 
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California State University, Fullerton 

Fullerton, California 92634 
SS 


DELPHI ROUND 5-A s 
Department of Philosophy SELNI Rey 90 
(714) 773-3611 Feb. 28, 1989 


Dear Delphi Colleagues, 


One of you wrote, "I’m beginning to think that to have done all that [we 
have done] and not have tried to define CT may turn out to be a mistake." 


Also in response to ROUND 4 three or four of you commented like this: "I 
have no major quarrel with any parts of your organization, [but] your 
emphasis leaves out a major component of CT ~- the dispositional 
component and the set of values inherent in being a critical thinker... I 
think it is a good working model of CT skills, but an incomplete picture 
of being a critical thinker." 

In view of the many positive responses to ROUND 4, yet sensitive to the 
concerns raised by comments like the above, ROUND 5 begins by building on 
Our success in articulating a decent first draft list of CT skills. In 
this letter we start right in on the question: For purposes of aer ral 
education assessment at the college lower division level, what do ~e 
experts recommend be included as a core critical thinking skill? 


In addition ta asking your endorsement of a revised draft of the skills 
dimension of CT, this letter also shares some key ROUND 4 results and 
some of the many useful comments you sent. 


The next letter, ROUND S-B, works on the two other aspects of CT you 
commented on in your ROUND 4 responses -~ namely CT's dispositional and 
the normative dimensions. Some of this also finds its way into the 
revised skills statement -- see 6.1! ROL. S-C picks up the remaining 
pieces of ROUND 4 and asks your approval of an outline of our report to 
the AFA Committee on Pre-College Philosophy. 


The table at the end of this letter shows that over 85% of us, (23 of 
26), could be described as fundamentally in accord with our first listing 
of CT skills. The second draft you are now being asked to consider 
endorsing was prepared in view of the many helpful comments and 
suggestions you sent in. I am very optimistic about the revised 
statement of CT skills, first hecause ROUND 4’s draft was approved by 
such a solid plurality, and second because your suggestions helped me 
substantially strengthen and enrich that statement. 


Since we are very close on so many things, your approval or disapproval 
of the expressions of our views presented in ROUND 5 should clarify 
things enough for me to start putting together our report to the APA 
Committee on Pre-College Philosophy. Where we have consensus our report 
will say so. ‘Where we diverge, it will say that as well. 


I truly appreciate all that you have already contributed, and I realize 
you you are all very busy folks. Yet I beg your continued indulgence. 
Please respond to the three round 5 letters with all reasonable dispatch. 
All responses are welcome, no matter how brief or selective. 


With sincerest gratitude, 


to, 
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Half of this letter shares comments regarding defining and testing CT 
made in response to ROUND 4. Before digging into our revised statement 
of CT skills, you might jump to page & and look through the comments or 
examine the tabular results of our earlier work on page 10. I learned, 
for example, that not listed in our original statement was a skill the 
majority feels is part of CT -- arguing. And, given what this Delphi 
project is all about, how on earth could I have omitted the CT skill of 
analyzing an idea from the first draft? 


ROUND 5-A, PART 1 -~ CT SKILLS 





INSTRUCTIONS: Consider the following amended description of CT skills along 
with the accompanying statements. Starting with the title and preamhle, make 
any needed changes, deletions, or additions (editorial or substantive). After 
working through the descriptions and statements you will be asked specific 
questions regarding endorsement. Please respond to those question as well. 


Skills Dimensions of Critical Thinking 





For purposes of general education assessment at the college, lower 
division level, we understand CY to include the cognitive skills of 
interpreting, inferring, analyzing, evaluating, expressing and monitoring. 
Because of our collective conviction regarding their centrality to CT, we urge 
those persons interested in assessing the skills dimensions of CT focus on 
these six abilities. However, since CT can be subclassified in a number of 
legitimate ways, our subclassification should not be interpreted as an 
educational taxonomy nor as implying or presupposing any psychological, 
Jogical or epistemological order or sequence of skills. While including those 
skills we take to be central to CT, we do not claim’ that our list is 
exhaustive in either breath or detail. 


Critical thinking involves actively interpreting one’s experiences and 
self-consciously making and expressing one’s analytical, evaluative and 
inferential judgments regarding what to believe or do. As such, critical 
thinking is a pervasive and multi-dimensional human phenomenon involving both 
dispositions and skills. Without diminishing the vital importance of 
cultivating CT dispositions throughout the K-12 and post-secondary educational 
process, we have here chosen to focus our attention on listing and describing 
CT abilities. As a goal statement of what a generally educated college level 
critical thinker should be able to do, we hope our consensus description of CT 
skills will assist in CT assessment and CT curriculum development both at the 
college and the K-12 levels. 


Among the nany ways one might improve one’s CT are by reflecting on one’s 
reasoning processes and learning how to think more analytically, objectively 
and logically, by expanding one's repertoire of those wore specialized 
procedures and criteria used in different areas of human thought and inquiry, 
and by increasing one's base of information and experience. An implication of 
our analysis of CT skills, however, is that CT skills per se transcend 
specific disciplines, yet executing them successfully in certain contexts 
demands background knowledge, some of which may be specific to how one makes 
reasonable judgments in that context. Since becoming adept at CT involves 
learning to use CT skills effectively in many different contexts we cannot 
overemphasize the value of a solid liberal education to supplement the honing 
of one’s CT skills and the cultivating of ong’g CT dispositions. 


CG ~ 
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Haaes of Core CY Skills and Sub-skills 


1. Interpreting - Categorizing, Investigating, Decoding, Clarifying. 
2. Analyzing - Analyzing Ideas, Identifying Arguments, Analyzing Arguments. 
3. Evaluating - Assessing Claime, Assessing Arguments. 


4. Inferring - Querying, Conjecturing, Concluding, Developing Reasons. 
5. Expressing ~ Stating Results, Describing Procedures, Stating Arguments. 
& Monitoring - Self-examination, Self-correction. Í 


Descriptions of Core CT Skills and Sul-skills 


1. INTERPRETING: To comprehend the meaning or explain the significance of a 
wide variety of experiences, situations, data, events, judgments, conventions, 
beliefs, rules, procedures and criteria. 

1.1 CATEGORIZING: To formulate categories, distinctions, frameworks or 
questions, and to describe experiences, Situations, beliefs, events, etc., so 
that they take on comprehensible significance or meaning, as for example to 
recognize a problem and define its cnaracter without prejudice to inquiry. 

1.2 INVESTIGATING: To actively seek, attend to, discriminate and describe 
experiential input relevant to a given situation, problem or concern; to 
gather input that conveys or is intended to convey data, information, or 
inferential relationships, as for example to gather evidence relevant to 
solving a problem in the light of how that problem is defined. 

1,3 DECODING: To actively detect, attend to and correctly understand, 
the informational content, effective purport, directive functions, intentions, 
purposes, symbolic significance, values, views, rules, procedures, criteria, 
or inferential] relationships expressed by others in convention~based 
communication systems, such as in language, social behaviors, ““a.tings, 
numbers, Signs and symbols, 

1.4 CLARIFYING: To explain, paraphrase or make explicit, through 
stipulation, description, analogy or figurative expression, the contextual, 
conventional or intended meanings of words, ideas, concepts, statenents, 
behaviors, drawings, nunbers, signs, synbols, rules, events or ceremonies; to 
an extent proportionate with the purposes at hand, to use stipulation, 
description, analogy or figurative expression to remove confusing, unintended 
vagueness and ambiguity, or to design a reasonable ;rozedure for so doing. 


2. ANALYZING: To identify the intended inferential relationships anong 
statements, questions, concepts, descriptions or other forms of representation 
intended to express beliefs, judgments, experiences, information, opinions. 

2.1 ANALYZING IDEAS: to identify expressions used in communication and 
determine the role they are intended to play in arguing or persuasion, as for 
example to identify a phrase intended to trigger a sympathetic emotional 
response and induce an audience to agree with an opinion; to identify related 
judgments, views, or concepts and to determine the conceptual similarities and 
differences between them; to identify issues or problems, deteraine their 
component parts, and identify the conceptual relationships of those parts to 
each other and to the whole. 

2.2 IDENTIFYING ARGUMENTS: Given a set of statements, descriptions, 
questions or representations, to determine whether it does express or was 
intended to express a reason or reasons in support of or contesting sone 
claim, opinion or point of vien, 

2.3 ANALYZING ARGUMENTS: Given the expression of a reason or reasons 
intended to support or contest some clain, opinion or point of view, to 
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identify: (ai the intended main conclusion, (b) the premises and reasons 
advanced in support of the main conclusion, (c) further premises and reasons 
advanced as backup or support for those premises and reasons intended as 
supporting the main conclusion, (d) additional unexpressed elements of that 
reasoning; such ae interaediary conelusions, unstated acsuestions or 
presuppositions, (e) the overall structure of the argument or intended chain 
of reasoning, and (f) any items contained in body of expressions being 
examined which are not intend tc be taken as part of the reasoning being 
expressed or its intended background. 


3. EVALUATING: To assess the credibility of statements, descriptions, 
questions or other representations expressing experiences, situations, 
beliefs, judgments, or opinions; and to assess the strength of the expressed 
inferential relationships among suth statements, descriptians, questions or 
other forms of representation. 

5.1 ASSESSING CLAIMS; To assess the degree of credibility to ascribe to 
a source of information or opinion; to a:. se the relevance of questions, 
information, principles, rules or proceduss: ;,ections to a given issue or 
concern; to assess the truth or the level 3 > > idence to place in any given 
representation of an experience, Situation, judgment, belief or opinion. 

3.2 ASSESSING ARGUMENTS: To deteroine the nature and quality of 
expressed inferential relationships; to judge whether the assumed truth of the 
premises of a given argument justify one’s accepting as true, or very probably 
true, the expressed conclusion of that argument; to anticipate and raise 
questions and objections, and then to assess whether these point to 
Significant weakness in the argument being evaluated; to determine whether an 
argument relies on false or doubtful assumptions or presuppositions to judge 
how crucially these affect its strength; to judge between reasonable and 
fallacious inferences; to judge the probative strength of an argument’s 
premises and assumptions with a view toward determining the acceptability of 
the argument; to determine and judge the probative strength an argunent's 
intended and unintended implications with a view toward judging the 
acceptability of the argument; to judge the extent to which additional 
information would strengthen or weaken an argument. 


4; INFERRING: To identify and secure elements needed to make inferences and to 
determine the inferential relationships between or flowing from statements, 
descriptions, questions, or other forms of representation on the basis of 
which inferences can be drawn, 

4.1 QUERYING: to recognize the need for evidence or information of some 
kind, in particular to recognize which statements, including those offered as 
premises, need justification, and to formulate and execute a reasonable 
strategy tor Seeking and gathering that evidence or information, 

4,2 CONJECTURING: Given a problem, question or point of view on an 
issue, to formulate multiple alternatives, develop hypotheses, or postulate 
suppositions, and to design reasonable strategies for determining their 
plausibility, viability or relative merit; to objectively draw out the 
presuppositions and the consequences of decisions, positions, beliefs or viens 
with which one might agree or disagree. 

4.3 CONCLUDING: Given a set of statements, descriptions, questions or 
other foras of representation, to educe with the proper level of logical 
strength, their inferential relationships, both deductive and inductive, to 
educe the consequences or the presuppositions which they support, warrant, 
iaply or entails; to successfully eaploy var’ ous sub-species of inductive or 
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deductive reasoning, as for example to reason analogically, arithnetically, 
dialectically, scientifically, etc. 

4.4 DEVELOPING REASONS: Given a question to be answered or a position on 
an issue, use appropriate inductive or deductive modes of inference to 
articulate reasons for answering the question one way as opposed to another, 
or for supporting or for opposing the position. 


Ss: EXPRESSING: To state, describe or represent to one’s self or to others the 
results of one’s reasoning and the way one went about producing those results. 
5.1 STATING RESULTS: To produce accurate statements, descriptions or 
representations of the results of one's reasoning activities so as to analyze, 
evaluate, infer from, or monitor those results, or so as to accurately and 
effectively recall or represent those results to one’s self or to others. 

5.2 DESCRIBING PROCEDURES: To represent as clearly as possible how one 
came to one’s interpretations, analyses, evaluation’or inferences, so that one 
might accurately record, evaluate, describe or justify those processes to 
one’s self or to others, or so as to remedy perceived deficiencies in the 
general way one executes those processes. 

5.3 STATING ARGUMENTS: To present arguments which communicate one's 
grounds for accepting some claim, their Jogical force in supporting that 
claim, and, as necessary, meeting objections to the premises one relied on or 
the reasoning one employed. 


6: MONITORING: To self-consciously regulate one's cognitive activities, the 
elements used in those activities, and the results produced by those 
activities, particularly by applying analyzing and evaluating to one‘s/own 
inferring with a view toward confirming, validating, correcting or questioning 
either ane’s reasoning or one’s results. 

6.1 SELF-EXAMINATION : To reflect carefully on one’s own reasoning and 
verify both the results produced and the correct application and execution of 
the cognitive skills involved; to make a thoughtful meta-cognitive self- 
assessnent; to reflect on the extent to which one’s thinking is influenced by 
deficiencies in one’s knowledge, or by stereotypes, prejudices, emotions or 
any other factors which canstrain one's objectivity or rationality; to reflect 
on one’s motivations, values, attitudes and interests with a view toward 
determining that one has endeavored to be unbiased, fair-minded, thorough, 
objective, respectful of the truth, reasonable, and rational in “soning to 
one's interpretations, analyses, evaluations, inferences, or expressions. 

6.2 SELF-CORRECTION: Where self-examination reveals errors or 
deficiencies, to design reasonable procedures to remedy or correct, if 
possible, those nistakes and their causes. 


H#HEHHE RE 


ROUND S-A, PART 2, == ENDORSEMENTS. 


1) Do you endorse the above statement as useful for purposes of 
assessing the skills dimension of CT at the lower division college 


levei? 


2) Would you be willing to bave your name listed in association 
with the above description of CT skills as a contributing member 


of the Delphi research project which generated it? 
KKK 
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ROUND S-a, PART 3 -- Reactions 


- 
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The responses to ROUND 4 included sone telling observations, thoughtful 
objections and well focused criticisms. Sharing these with others in 
the Delphi group is essential to the Delphi process. J've included as 
many aS 18 feasible and as can he understood outside the context of 
whatever ROUND 4 item may have prompted then, father than use of your 
time reading positive comments, of which there were many, I've stuck 
chiefly to the critical anes. 


INSTRUCTIONS: Read and consider what our colleagues are telling us. In 
addition to the adjustments already incorporated into the above draft 
statements regarding CT and CT assessnent, what other responses and 
improvements should we make? In the jight of these comments and other 
concerns that come to nind, what specific recommendations should we 
include in our report to the APA Cossittee on Pre-College Philosophy? 


SELECTED RESPONSES OF CT TESTS AND TESTING CT 





t “A test cannot be considered in the abstract, without working 
out its intended use and intended users, the specific population to be 
tested, and the discriminations the test would be required to make... 
The domain the test covers is governed by this context, and concepts 
that constitute svbslassifications of the domain are arranged in 
different ways from the ways in «hich they might be arranged for a test 
with the sane name but a different purpose... CT can be subclassified 
in a number of legitiaate ways, with many of the same elements 
recurring in different places in different classification schenes..." 


+ "There's hardly anything we now need more than improved CT 
tests. I’ve used... in pre-post testing for several years. It's the 
best I know of. (CBut...J" 


t ",,.to make sense of CT we must sake sense of the critical 
thinker. The cognitive must be discussed in relation to the affective, 
and both must be discussed in relation to their roles in the real 
world. Curriculum and assessment rust be put into some broad context. 
... There is no one right definition of CT, and in testing we should 
never confuse testing for aicro-skills with testing for CT itself. 
Host CT tests are aicro-skil] tests only. They are valuable, but only 
in a qualified way. The Delphi project seens wel] on its may toward 
confusing the part with the whole.” 


+ "The {Round 4 list of skills] seess fine to ne, Ynere are 
definite limits to armchair analysis, and until soaeone actually starts 
trying to aeasure these things, it is dafficult te know just how t9 
revise the list of CT skills and sub-skills.” 


t “Please interpret ny responses cautiously. I have not spent 
much time trying to define CT. My business is trying to do] 
assessment once you guys arrive at clear and userul definitions!” 


e “We sust focus on the purpose and target audience of an; given 
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CT test. Only that way can we fine tune our definition of CT, or 
should I say our "theoretical construct" of CT.” 


* "Rather than limit our conception of CT to achieve a certain 
kina of test, why not simply make more modest claims about the test? 
Why not say that yau are testing aspects of CT? This Seems nore 
justified. Otherwise the test becomes the tail that wags the dog... Be 
honest. about what we are and are not testing by multiple choice tests. 
Let's not reduce rich and complex realities like CT to that which can 


be directly tested in the multiple-choice format.” 


# [Some of your descriptions] are difficult to test without open- 
ended items... [For example] it's difficult to test formulation of 
strategies, MC exams focus more on recognition of best strategy., 


* "The categorization system...gives little guidance on what 


. precisely to teach and test. For instance, under "evaluating" is "to 


assess the credibility of statements." But what should be taught when 
doing this, and what should be tested when trying to find out what 
skills students have? If teachers are supposed to act based on what we 
produce, I believe we need to inelude criteria for assessing 


“credibility. Again I refer you to..." 


e "CT is deeply connected in those individuals who are successful 
critical thinkers to a pervasive self consciousness about one’s Own 
thinking and reasoning processes. Such self consciousness should be 
deliberately cultivated in our students and should therefore be 
included Somewhere in the overall description -- it is an intrinsic 
part of the {CT] process. It monit' s the selection, application and 
interlinkage of the various relevant processes." 


+ "Until we have a theory of reasoning (a combined nornative 
theory of informal logic along with a descriptive theory of cognitive 
processes) we will not be able to spell out CT skilis non-arbitrary 
with completeness and pretision.... We are proaducing...a frasenwork, 
{that is] a list of concepts used for understanding a domain. If that 
is so, it will play hell out of our attempts to assess CT skills. l 
doubt that we will be able to manage construct validity for any 
conventional MC test with our [list of CT skills]. Yet, what we are 
coming up with is extremely valuabie if we focus on performance 
assessment. Our framework gives us a passably good set of criteria by 
which one would judge goud performance on CT tasks. The criteria are 
developed by experts ~- us -- and we are currently judging whether we 
accept them or reject them, another step in the process of developing a 
good performance assessment. The next step would be to distribute 
typical essays, (good, bad and ugly ones) and aim at some consensus in 
telling the good ones from the bad ones." 


# "We should resist the assertion that CT is domain dependent." 
€ "I'a beginning to think that to have done all that (we have 


done] and not have tried to define CT aay turn out to be a aistake... 
What aakes (the list of] inportant, indeed basic, intellectual skills 
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Cin Round 4) a list of critical thinking skills? It could just as 
well be a list of rationali thinking skills, or logical thinking skills 
or higher order cognitive skills. ...] fail to see how this list 
captures the force œf the word "critical". CEtymologically) the 
meaning af "“criticai" is judging, evaluating, estimating the worth of 
Something... A critical thinker is someone nhu renders an opinion on an 
intellectual product...by assessing the strengths and weaknesses of 
that product... Doing sec requires the capacity to elicit and apply 
standards, principles and criteria. None of the [Round 4] list is 
really this skill. If we asked for a list of problem solving skil?s, 
would we get the same list fas in Round 4]? If ss, then either there 
is no conceptual difference between the two, which I think is wrong, or 
else the list fails to capture what is distinctive about CY skills. CT 
also connotes "crucial". (Here) the (Round 4) list fares better, 
because...these six skills are crucial --i.e., essential for 
intellectual survival. The problem is that the list is so bread and 
wide-ranging that it is not clear what intellectual skills have been 
excluded." 


+ "Tf one identifies CT as that which makes a critical person to 
be what he is, then [your] narrow concept of CT is inadequate. GT 
iS... an answer to the general problem of conformity, prejudice, 
narrow-mindedness, and irrationality in the world. CT is what one does 
to achieve autonomy and independence of thought, to lesson one's 
prejudices, to broaden one’s perspective, and to become more rational." 


+ "The main overall worry is that the categories are auch too 
broad. When we get down to testing it will not be for something like 
"assessing logical strength” or "clarifying" but very specific skills 
such as "recognizing whether sonething is a necessary or sufficient 
condition; recognizing the difference between if p, then q's and if q 
the p's etc. The lines of demarcation are very unclear e.g. between 
psychological end logical (epistemological) criteria conrerning say 
observation." 


+ "I have no major quarrel with any parts of your organization, 
[but] your emphasis leaves out amajor compcnent of CT ~-~ the 
dispositional component and the set of values inherent in being a 
critical thinker... I think it is a good working model of CT skills, 
but an incomplete picture of being a critical thinker." 


* "There is no attention... to the dispositions... Characteristic 
of CT, These... areas essential to CT as are the knowledge and 
information used in the processes.” 


+ "The dispositions are at least as irportant as the skills. I 
suggest their addition," 


+ "CT works by recognizing and criticizing sources of inforeatiaon, 
by drawing implications from given materials, identifying assuaptions, 
noticing relationships of consistency, inconsistency, implications and 
contradiction, inferring interesting consequences, recognizing, 
analyzing and evaluating arguments and constructing them as well. Of 
course, there's a lot more to it.” 
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* "Arguing is not listed as a separate skill!” 


* “No mention is made of understanding another's purpose... this 
would fall under interpreting, I suppose." 


£ "I disagree with the list of CT skills as described. 
Interpreting is obviously a cognitive operation. But it is not a CT 
operation. Expressing is essentially low level] communication, not 
generative in the sense of CT. Monitoring is ‘meta-cognitive. The 
problem is that this description is so broad "critical" thinking gets 
lost in all the other kinds of thinking. This blurs the nature of CT 
beyond recognition... CT is "judging the worth, accuracy or 
Significance of something.” 


* "There are several items I mess overall. They may be subsumed in 
some of the processes you have listed, but very few people will be 
conscious of them unless they are brought out explicitly. One is the 
Capacity for arithmetical reasoning with ratio and division -- it 
begins with word problems in Sth and th grade arithmetic and carries 
up to exactly similar reasoning with concepts such as density, 
composition, contraction, rates of change, in more sophisticated 
settings. It includes the ration reasoning that goes with scaling 
areas, volumes rates, etc. This capacity is profoundly important in 
any CT that involves numerical information (whether it be scientific, 
economic, sociological, psychological... I'm talking about arithaetic 
and not satheaatics at the level of calculus or even algebra. CA 
second capacity to include is] "correlational reasoning.” Finally I 
miss explicit inclusion of the process of translating symbols (e.g. 
graphs, numerical data, histograms) into words or words into 
corresponding symbols. Such translation is essential to much CT.” 


+ "(The concept of CT should also include) discriainating 
explicitly between the factual or experiential input and the inferences 
drawn, identifying gaps in available information and identifying 
irrelevant or superfluous information, and it should include the 
Capacity to cansider some situation in the abstract and, by applying 
relevant governing principles or constraints, arrive at reasonable and 
plausible conclusions about the outcomes that would result fron the 
imposition of some change =- hypothetico-deductive reasoning." 


* "Include something about cppositional reasoning -- taking the 
part of one with whom one diSagrees. Also include hypothetical 
reascning ~- reasoning from suppositions and hypotheses. 

+ "Taxonomy" is not a good word for what we’ve got; it’s sore like 
a list. Having a taxonomy in biology and education is to have a 
hierarchical set cf categories such that each subsequent step in the 
hierarchy subsumes tne steps below it. We don't have that here. 

® "“(Conjecturing] is creative thinking not CT, 


* “UTo have included seeking and gathering evidence] is anbiguous. 
We are not scientists. 


e "(Regarding expressing,] I don't see any strong rationale for 
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extending the focus to communicating the results of CT. 
t "None of [your siJ listed items are skills -- let alone sub= 


skills. They are general categories into which many (at least dozens 
af) distinguishable skills may be lumped. This as important, 
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Agree Disagree Unsure 
1. INTERPRETING 24 = 2 
Observing 20 3 3 
Decoding 235 = 3 
Clarifying 26 ae - 
2. INFERRING 25 = 3 
Querying 24 1 1 
Conjecturing 23 2 1 
Drawing Conclusions 24 = 2 
3. Analyzing 23 - 3 
Locating Arguments 23 2 
Parsing Arguments 23 - 3 
4. Evaluating 24 - 2 
Verifying Claias 25 = ! 
Assessing Logical Strength 26 - - 
5. Expressing 21 3 3 
Stating Results 21 3 5 
Describing Procedures Zl 3 S 
ó. Monitoring 24 - 2 
Regulating 2 = 3 
Reviewing 23 = 3 
Correcting 24 - 2 


RESPONSES TO "WHAT CT IS HOT” 
To help delineate CT through conparisons and contrasts, I offered a list of 
activities that bore sone family resemblances to CT. Each of them depends on 


CT. Sut whether any of was CT per se was the issue. Here's what you said: 





Isn't Is Partly 

cI T a 
1, Sensing, (Seeing, Touching, Hearing, etc.) 16 2 5 
2. Reading, Listening, 13 4 6 
3. Speaking or writing, 14 3 6 
4. Motivating, persuading, selling, 14 4 5 
5. Interrogating, cross-examining, petitioning, 12 6 6 
ó. Fhysically investigating the world around, 13 4 6 
7. Trouble-shooting, problem-solving, puzzle solving, 13 4 6 
8. Decision-making, selecting, choosing, exercising will, 14 3 6 
9. Planning, defining goals and objectives, 10 7 6 
10. Finding or ascribing a *eaning to art, life, or events, 13 5 6 
11. Defending an opinion or belief, arguing a case, 7 7 8 
12. Managing, administrating, governing persons or things, 16 2 5 
13. Philosophizing, 12 3 8 
14. Conducting research within any particular discipline, 12 3 8 
15. Experiencing, feeling, emoting, or empathizing, 15 3 5 
16. Communicating using language. 15 2 6 
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California State University, Fullerton 
Fullerton, California 92634 





Department of Philosophy 
(714) 773-3611 


DELPHI ROUND S- 





Mar. 6, 1989 
Dear Delphi Colleagues, 


This letter follows the outline in the Round 5-A letter -- 
questions first, background information second. The focus here is on 
fleshing out our conceptualization of CT beyond the revised list af CT 
abilities presented in Round 5-A. Specifically this letter responds to 
your comments regarding those dispositional and normative dimensions 
many include when describing CT. The implications for K-12 and college 
level assessment, curriculum development and pedagogy of including 
either of these dimensions are crucial for our effort. You're asked for 
your ideas about this, toa. 


To get things started, on the next nage you'll find a draft 
statement regarding the dispositional dimension of CT. Following that 
is a draft statement regarding the normative dimension. These two draft 
statements are based on your comments regarding needing a fuller 
conceptualization of CT and an analysis of commonly referenced concepts 
of CT which appear in the literature. Have at those draft statements. 
Amend, edit, comment, accept, reject ~- whatever you think will help us 
be able to present an intellectually credible and educationally useful 
conceptualization of CT. 


Coming soon: Round S-C focuses on our report to the APA Pre-College 
Philosophy Committee. It will include a proposed outline of that. 
report, showing how the various pieces of the assessment puzzle we've 
worked on for over a year now will be incorporated. It also picks up 
the two pieces from Round 4 which haven't been addressed yet, namely the 
input-output model Cwhich we rather roundly rejected], and the statement 
of what a cognitive skill is and how cognitive skills might be assessed. 


I'll be speaking on March 24 at 1:00 p.m at the Pacific Division 
meetings of the APA in Oakland CA, sharing a little of what we've been 
doing and the direction things seem to be taking. If you happen to be 
in the neigtiborhood, please stop by so we can visit. 


Knowing you are very busy, I beg mercy and ask you to reply as soon 
as is reasonably possible. A quickie note with general comments is 
helpful, if you won't have time to go through things in careful detail. 


Thanks for your continuing support and involvement. 


Yours sincerely, 


a 
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Pete Facione 








Part i: The Dispositional Dimension of CT 
INSTRUCTIONS Reflect on the following statement. Edit, amend, or 
revise as you see fit. Some background information is presented in the 
second half of this letter. Please respond to the following questions: 

1) Should our final report include a statement on the dispositional 
dimension of CT? Why, why not? 

2) What implications or recommendations for K-12 and college lower 
division level assessment, curr.culum development and pedagogy follow 
from including a dispositional element in our conceptualization of CT? 

CT -- The Dispositianal Oinension 


In addition to Its cognitive skills diaenston, CT also involves certain 
cagnitive dispositions, personal traits or intellectual virtues which are 
crucial to its broad and successful use, Just as nith the cognitive 
abilities dinension of CT, when conceiving of CT assessaent or an 
Instructional program in CT, it is isportant to consider ways of developing 
paterials, teaching strategies and assessaent tools which focus on CT 
dispositions, The cultivation af CT cognitive dispositions is particularly 
Important in Insuring the use of CT abilities outside the narron CT 
Instructional setting. Persons nho have developed the intellectual virtues 
and personal traits listed belon are auch aore likely to apply their CT 
skills than are those who know the skills but lack the intellectual 
discipline toa use thea, 

The critical thinker is one who possesses and cultivates the cognitive 
dispositions, personal traits or intellectual virtues af 

* seeking a clear statesent af issues, questions or probleas, 

* curiosity in exploring issues and seeking information, 

k eagerness in seeking and in employing CT abilities, 

* openness toa seek and to consider divergent viens or alternatives, 

> trust in the processes af reasoned inquiry, 

® perseverance, diligence and discipline in keeping well-inforned, 

* hanesty and huaility tn appraising one’s ann reasoning and viens, 

® willingness to seek, use and mention credible sources, 

x prudence In suspending or waking judgwents and in taking ar Changing 
one's position, 

x wisdow and persistence in the use of appropriate, defensible and 
relevant criteria, 

š e. ort ta address issues reasonably and to remain relevant to the 
basic concern or problen, 

* precision, to the extent the subject peraits, 


x orderliness in the treatuent of conplex issues and processes, 
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Part 2: The Normative Dimension of CT 
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INSTRUCTIONS Reflect on the following statement. Edit, amend, or 
revise as you see fit. Some background information is presented in the 
second halt of this letter. Please respond to the following questions: 

1) Should our final report include a statement on the normative 
dimension of CT? Why, why not? 

2) What implications or recommendations for K-12 and college lower 
division level assessment, curriculum development and pedagogy follow 
from including a normative element in our conceptualization of CT? 


CT -- The Harnative Diaensian 

In addition to its cognitive skills and dispositional diaensions, èT 
also involves certain normative features which gavern its proper use. 
Understanding that in naking this Statement we are going beyond a purely 
descriptive analysis, ne judge it vital to include this naoraative conponent 
in aur conceptualization of CT. In a free and rational society education 
aust be pore than skills training and aore than the inculcation of a set af 
personal dispositions. Education aust include the nurturing af those civic 
and persanal values which insure that the heritage of intellectual fair- 
aindedness and political freedoa wild be passed to future generations. 

Properly used, CT contributes to the fairmainded analysis ane resolution 
of issues, rather than the blind, sopnistic, or irrational defense of viens 
known to be intellectually defective or biased. 

Properly used, CT unites thase who would reason together objectively in 
a reflective and intellectually Saund process, even if their final judgnents 
and analyses should disagree. 

Properly used, CT prosotes rational autonoay, intellectual freedoa, and 
the objective investigation af any issue or cancern whatsoever. 

| Properly used, CT treats all affected persons with sensitivity and with 
equal respect and dignity ~-~- it is never exclusively self-interested, 
abusive, coercive, or without integrity and gaad faith. 

Thus, in a free, open and rational society, when conceiving af CT 
assesspent or an instructional prograa in CT, ane aught to consider ways af 
develaping saterials, teaching strategies and assesssent tools which Insure 
the noraative dimension af CT will be developed in students, in addition ta 


CT skills and dispositions, 
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Part 3 --RACKGROUND 


For your consideration, here are the views of six widely cited persons 
regarding what CT involves. 


1) CY is. ... "Active persistent and careful consideration of a belief or 
supposed form of knowledge in light of the grounds that Pappan it and the 
further conclusions to which it tends" John Dewey. 1909 


2) CT “is reasonable reflective thinking that is focused on deciding what to 
believe or doa." In addition to 12 CT abilities, CT also includes 14 
dispositions. Namely: "to seek a clear statement of the thesis or question, 
to seek reasons, to try to be well informed, to use credible sources and 
mention them, to take into account the total situation, try to remain relevant 
to the main point, to keep in mind the original or basic concern, tu look for 
alternatives, to be open-minded, to take a position when the evidence and 
reasons are sufficient to do so, to seek as much precision as the subject 
permits, to deal in an orderly manner with the parts of a complex whole, to 
use one's CT abilities, to be sensitive to feelings, level of knowledge, and 
degree of sophistication of others" Robert Ennis, 1987. 


3) "CT is an attitude of being disposed to consider in a thoughtful way the 
problems and subjects that come within the range of one’s experience; 
knonledge of the methods of logicai inquiry and reasoning; and some skill in 
applying \hose methods..." E.G. "recognize problems, find ways to meet them, 
marshal pertinent data, recognize assumptions use language well appraise 
evidence evaluate statements, see logical relationships, draw warranted 
inferences, test one’s own conclusion, reconstruct one’s beliefs based an 
experiance, render accurate judgments." Edward Glaser, 1941, 


4) "CT can be characterized as skeptical, radical, disillusioned, problem- 
seeking, holistic, judgment-oriented, non~algorithmic, constructive, 
comprehensive, empathic, meta-cognitive, higher-order, reflective reason- 
governed, logical, self-correcting, context-sensitive, criterion-referenced, 
crisis-oriented, normative, inferential, pragmatic, purposive, relational, and 
open- minded." Matthew Lipman, 1988. 


5) "A given student, S, is a critical thinker in some area requiring mental 
effort, X, if S has the disposition and skill to solve problems in X using 
some subset of ‘the available evidence E pertinent to activities in area X. 
"CT" has an identifiable meaning but the criteria for its correct application 
vary from field to field." John McPeck, 1981. 


6) CT is disciplined, self-directed thinking which exemplifies the verfections 
of thinking appropriate to a particular mode or domain of thinking. It comes 
in two forms. If thinking is disciplined to serve the interests of a 
particular individual or group, to the exclusion of other relevant persons and 
groupS,... it is sophistic or weak sense critical thinking. If the thinking 
is disciplined to take into account the interests of diverse persons or 
groups, it is fair-minded or strong sense critical thinking." Richard Paul, 
1988. 
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Here are your comments regarding statrment #4 in the caveats and cautions 
section of Round 4, "If CT ig a set of attitudes as well as a set of 
skills, that poses no problen for assessnuent because we can just develop 
nays to assess the CT attitudes too.” 


w= "Difficult to do!" 
-- "CThis] seems too facile. To be sure, rhetoricians and social 


scientists study attitudes and so have developed ways to ascertain or measure 


certain attitudes. But does this mean that all attitudes can be so tested? How 


can we talk about whether certain attitudes can be tested until we have 
enumerated those attitudes and so know specifically what we are trying to test? 
Also, can't some aspects of proficiency in CT, or beinn a critical thinker, 
involve both skill and attitudes combined? If CT involved recognizing one’s own 
prejudices and stereotypes, one must have an attitude of willingness to confront 
one's stereotypes and the skill to delineate and recognize them. Attitude and 
skil) would qo hand in hand. We don't just have a set of skills and as set of 
attitudes. This is too atomistic a description." 

-= "Agree" [with (6) as stated, no comment.) 6 people. 

-~ "Agree, we should teach skills and dispositions, the attitudes are 
minor,” 

-~ "Agree, but attitudes are harder to test." 

-- "Agree. CT is more a set of attitudes than a set of skills... It is 
easier to change attitudes in a single course than to improve skills 
Substantially. Changing attitudes and fostering self-monitsring should lead to 
lifelong improvement in a student's CT performance. So it’s actually very 
important to say s1mething about assessing CT attitudes..." 

-- "Disagree. The attitude/skill distinction isn’t that sharp.” 

= "How?" 

-- "Good luck!" 

-= "Unlikely." 

-- "There i5 a very great difference between skills and attitudes (although 
there are elisions at the boundaries of both concepts). Richard Paul has taught 
us most effectively (that] there is, or should be, a causal relation between 
Skill and disposition such that the disposition might/should elicit, motivate, 


and work as Side-constraints for the skills." 
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Here.are your comments regarding statement #7 in the caveats and cautions 
section of Round 4, "CT is thinking skills. Saying CT is a set of 
attitudes may be a way of describing what people who are good at CT are 
like, but it is not a may of describing CT itself. 


-- "Good point!" 

-- "CThis] simply expresses a position dogmatically, and in the light of 
discussions by Richard Paul, Harvey Siegel, and others this is just not 
acceptable. To dismiss these positions without argument is just not 
philosophical!... [This] simply rules out a major position on what CT is. 

-- "Agree" [with (7) as stated -- no comment]. 3 people, 

~~ "The attributes required for CT can be instilled given enough time and 
the right emphasis of subject matter in CT, but mot all CT subject matter 
instills proper attitudes. 

-- "Disagree" [with (7) as stated -- no comment]. 3 people. 

-~ "Disagree, (we) need to include some of the attributes," 

-~ "Good move. Go for the [CT] itself!" 

-- "Agree, but I often find that performance does not change until I get a 
change in attitude. That the attitude has changed is best seen in the improved 
performance... The most important attitude is "I really want to find out what 
you believe. and what your reasons for believing it are. Then when I am sure 
that you agree that I have understood you, I will think about and state where I 
differ from you." 

~~ “Agree, but our ultimate concern is to encourage people to think 
critically when it is appropriate to do so. And that requires a critical spirit 
(critical attitudes) as well as thinking skills and knowledge." 

-- "Disagree. The attitude/skill distinction isn't that sharp." 

~~ "Of course, some descriptions of dispositions are ways of describing 
what people who are good at CT are like, but I fail to see how it follows from 


this that they are not also ways of describing “" itself," 


ttt 
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Department of Philosophy 
(714) 773-3611 








Mar. 10, 1989 


Dear Delphi Colleagues, 


This letter asks you to consider three things: 
x (1) An outline of our report to the APA, (page 2), 
(2) A comment on skills and skill assessment, (page 3), 
(S) Specific recommendations we should make, (page 4) 


Your ROUND 4 comments on what a skill is and how a skill can be 
taught and assessed are very revealing. Take a look, for example, at 
the differences of opinion over the relative superiority of two of the 
strategies of assessment. The research Steve Norris is doing, whether 
or not idiot savants can be said to be skilled, and the basic conflict 
between practical efficiency and accuracy in assessment, all found there 
way into your responses. (See pages 5 and 6.) This is another example 
Of when an apparently off the wall question stimulated some useful 
ideas. 


Early responses to ROUND 5-A are coming in already. That's great. 


Please respond to the three round five letters as soon as is 
reasonably possible for you. I plan to start putting our report 
together this semester. Naturally you'll have a shot at it before 
final revisions. If all goes weil we'll be ready to submit our report 
to the Pre-College Committee in the fall. 


Depending on the quality of the recommendations that come, we 
could be very close to wrapping things up. Thanks for your continuing 
support and active involvement. 


Yours sincerely, 


A 


/ 
es 


Pete Facione 
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: Proposed Outline of Gur Report 


INSTRUCTIONS: Consider the following outline of our Delphi 
research report to the American Philosophical Association 
Committee on Pre-College Philosophy. Comment, make any and all 
additions, deletions, amendments, changes you think reasonable. 


CT -- A Theoretical Construct for Purposes of Assessment 
Submitted to the Pre-College Philasophy Committee 
American Philosophical Association 


EXECUTIVE SUHHARY 
A. Delphi Conceptualization of CT 
B. List of Delphi Recommendations 
C. Endorsement of Members of Delphi Research Panel 


REPORT 
I. Introduction 


A. The Concern for CT Assessment 
1. National interest and large scale assessment 
2. CT assessment in the inaividual classroom 
2. APA memoers concern, college and K~12 ‘level 
R. Formation o+ the Delphi Research Project 
1. Charge to the project director 
2. Description of Delphi Research Methodology 
3. Building the Delphi Panel of Experts. 


II. Delphi Findings, Points of Agreement and Disagreement 
A. Preliminary assumptions, CRounds 1 and 2) 
R. Conceptualization of CT, CRounds 3, 4, and 51 
1. CT -- the skills dimension 
2. CT -=- the dispositional dimension 
S. CT ~- the normative dimension 
C. General Comment on Assessing a Skill. [Rounds 4 and 5-C] 


, 


III. Recommendations 

A. General Considerations Regarding Educational Assessment 

1. Validity, Reliability 

2. Difficulty and Discrimninability 

3. Purposes of a CT assessment 

4. Characteristics of persons being assessed. 
B. Strategies for Classroom CT assessment 

1. Some questions to ask one’s self 

2. Assessment, pedagogy and curriculum development. 

3. Suggestions on putting together a classroom CT test. 
C. Comments on Large Scale CT assessment 


e 


IV. Appendices 
A. A Quick List of Purported CT Assessment Tools 
B. A CT Bibliography with emphasis on Assessment 
C. Delphi letters from each Round 
D. Response rates to each Delphi and other tabular data 
E. List of Delphi Panel of Experts 
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Part 2: General Comment on Assessing A Skill 








INSTRUCTIONS: This revision is based on you Round 4 comments. Please 
edit. You migh* first read your colleagues’ comments, on pp. 5 and ó. 


A skill is the ability to perfars certain processes or procedures nore or 
less well. Having a skill includes being able to do the right thing at the 
right tine. 3o, being skilled at sonething involves knowing, perhaps 
implicitly or without the ability to articulate this knowledge, a set of 
procedures and when to apply those procedures. It also involves having sone 
degree of proticiency in executing those procedures, Reflecting on and 
isproving one’s own skills involves judging when one is or is not perforaing 
well, and considering nays of Improving one's performance, 


It is generally thought that skills, particularly cognitive skills. can 
be taught in a variety of ways including making the procedures involved 
explicit, describing hon they are to be appiied and executed, explaining and 
modeling their correct use, and justifying their application. Teaching 
cognitive skills also involves exposing learners to situations where there are 
good reasons to exercise the desired procedures, judging their performance and 
providing the learners with constructive feedback regarding their proficiency 
and ways of iaproving it. Instruction aight start nith situations that are 
artificially sinple, but it should culainate in situations that are realistic 
and complex. Teaching is not everything.’ The learners must contribute a 
weasure of effort, attention, practice, desire, and self-monitoring. Teaching 
skills involves sotivating leavners to achieve higher levels of proficiency 
and, particularly in the case of CT, independence. Jt also nay involve 
coaching learners on how they can achieve those goals. 


Persons can be judged as being aure or less proficient in a given skill. 
The first nay of assessing is to observe the person perforaing the skill and 
aake a judgaent regarding the degree to which the person possesses the general 
skill gn question. A second way is to compare the outcomes (if any) that 
result froa executing a given Skill against sone set of criteria. A third way 
is to query persons and receive their descriptions of the procedures and 
Judgnents they are using as they perfora that skill, would use if they were to 
perfora that skill, or did use when they perforaed that skill. A fourth way 
Is to conpare the outcomes (if any) that result frox perforaing another task 
against sone set of criteria. where the perforaance of that other task has 
been shown to correlate strongly with perforaance of the skill af interest. 
Ho matter which may is used, it is Important to ensure that the test 
conditions foster an attitude in which the test-takers are disposed to use 
their skills as well as they can, and are not constrained or inhibited fron» 
doing sa, lt is highly advantageous to crass check the results of any ane way 
of assessment against the results of other ways. 
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Part 3: Recommendations 





INSTRUCTIONS: Considering what we know and don’t know about CT 


eel 








assessment today anu considering the direction our Delphi effort has 
taken us, what advise, suynestions, recommendations should we make 
regarding CT assessment in the classroom or larger scale enterprises, 
(e.g. Naticnal, state, district wide, college wide CT assessment)? 

Mention your intended audience, and be as precise about the 
recommendations as possible. [Ce.g. To ETS and ACT we should say, «3 
to teachers of primary grades we should say,...3 to professors who teach 
CT at the college level we shoauld say,...3 to the research community in 


cognitive psychology we should say,...J. 


To help you think through this most important matter of making 
recommendations, I've drafted some questions. You need not respond to 

- them. They're only to stimulate thinking. Rather, formulate specific 
recommendations. Oh! A plea for mercy: I have no staff, so please dan't 
ask that I search for things you might have once said about this 


someplace else.) 


1, What questions have you found it useful to ask yourself regarding your 
own assessing of CT in teaching or professional work? 

2, How might a college age person who possessed the CT abilities we have 
described be able to show that she had those abilities’ 

3. What about showing that she has the designated CT dispositions or 
normative attitudes, (should we decide to include those in our final report)? 

4. How might a person in elementary school, junior high or high school 
show that she has those CT abilities, dispositions or normative attitudes? 

S. What questions or tasks might you ask a group of persons to undertake 
if you wanted good evidence regarding which persons were better at CT than 
which others? , 

&. If you were doing a workshop for teachers at some grade level, what 
would you tel] then about CT assessment? 

7. If you were serving as a consultant to some organization which wanted 
to initiate large scale CT assessment, like a school district, university, or 
state department of education, what would your recommendations be? 

8. If a colleague and friend asked what you really thought about how CT 
can best be taught, learned and measured, what would you tell your friend to 
do or avoid doing? 

9. What makes you optimistic about CT assessment? 

10. What do you most fear about CT assessment? 
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a Here are some of your comments on the idea of a skill and how 
skills can be taught and assessed. 

* ",.. Your ways of teaching skills are conspicuous in their lack of 
explicit emphasis on educational values including the use of morally justified 
procedures in teaching. Similarly your ways of learning lack such 
educationally important activities as explaining, understanding and 
justifying.” 


* "Mental skills are no less directly observable than physical skills. 
Skills are abilities, and abilities can't be observed. Thus, observing the 
skili as it is being perforsed is impossible for any skill.” 


* "I have my doubts about ... the implicit mind/body dualism. 
Performances can be measured directly (e.g. by observing a surgeon's technique 
and results) and indirectly (e.g. by a paper and pencil test of knowledge of 
anatomy). Both constitute a sample from which inferences are made to the more 
general skill, and this is all we need to concern ourselves with..." 


# "Your description exclude any effort, visualization, mental rehearsal, 
or using of any inner resources a person brings to skill learning and skill 
performance ~- concentration, automaticity, goal directedness etc." 


* "C€The narrative] presupposes that if one has a skill, one consciously 
knows the intricacies and interworking of that skill. Often this is simply 
not the case. ... Indeed idiot savants may be said to possess skills with 
little or no understanding of how they perform skilled activities. ... (Those 
who] understand Chow they arrive at the correct result] are better critical 
thinkers Cthan those arrive at the result) with no real understanding of the 


rationale. 


* ",..Don't we observe performances which we interpret as being skill ful 
or being evidence of skill at a certain degree or level." 


* "It seems odd to talk about performing a skill, given the initial 
equating of a skill with the ability to do something well." 


### (The second way is superior to the third...) 

"because the third way can be employed only after it has been 
settled that the person has the skill, and that can be settled only by 
comparing outcomes against criteria, i.e. by employing the second way. In 
short the third way is parasitic on the second. Beside, a skilled person may 
not even be able to describe the procedures and judgments used when executing 
the skill." 

“for the assessment of average people, because (1) CT skills are 
generally employed unconsciously so the third way just won't work for average 
people, and (2) it's fast, cheap and yields unambiguous (if not wholly 

ve trustworthy) answers... But the third way is Superior when dealing with 
experts, whom we can expect to be conscious of their procedures. If we can 
get an account of their actual thought processes, then we can assess those 
a processes directly rather than indirectly." 
7 “if instead of outcone we also include how the answer is arrived at; 
"i thus an idiot savant can produce answers to complex equations but not by any 
3 process we understand (or she does); if we liait ourselves to outcomes in 
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judging skill, we'll miss the distinction between the highly skilled 
arithmetic operator an the idiot savant. Since I would want to claim that the 
idiot savant is not a case of skill at all, this presents some difficulties." 

"because performance is what we want, provided that eutcones include 
the steps taken. That is, if a student makes a good judgment on Someone's 
reasoning, I would want to know how he/she got to that judgment." 

"because it iS much easier to Carry out in practice, provided that 
there are enough items to compensate for accidental correct and incorrect 
outcomes...” 

“because it may be less time-consuming, and because it would be 
easier to design instruments for..." l 

"because it does not depend cospletely on the testimony of the 
individual about what occurred. That’s a strength because the literature on 
self-reports suggests enormous difficulties with the agent's own accounts of 
their activities. If we have the product, and the account of the process 
which yielded it, we’re in better position to judge the degree of skill." 


x (The second way is not superior to the third...] 
"because the second is a prerequisite for carrying out the third; 
that is, to do a credible job of querying a person must fist have completed a 
strong comparison and examination or the argument” 

. "because, as far as CT is concerned, [the second] does not reveal 
the understandings and reasoning behind the answers given, but just assesses 
overt answers against some criterion. Clearly one could come to "wrong" 
answers depending on his understandings of the Situation, and yet this 
reasoning not be a violation of, or deficiency in, CT. Suppose someone asks 
how many piles of beans we may divide 27 beans into, if we put 5 beans in each 
pile. Suppcse someone says 6, because he allows that He Can have small piles, 
the remainder can constitute a pile in its own right. Given his 
understanding, he has given the "right" answer. Does having his understanding 
mean he is not thinking critically? ...Qbviously, the third way has its 
drawbacks, since it is far more consuming than checking outcomes against a set 
of criteria. A truly superior method of testing CT would incorporate both." 

"because there are often other possible explanations of the 
outcones." 

"because the outcome could have been accidental or a consequence of 
good luck, or bad luck, or in the case of testing, copied." 

“because the third method might reveal that a person has good CT 
skills where the second method suggested poor skills.” 

"Cif one cannot specify) the criteria against which to judge the 
process~outcome," 


+ "I do not regard the second and third ways [of assessing] as 
constituting a hierarchy with one way superior to the other. I use both ways 
in assessing Students and check the two against each other, whenever I can. 


e "Part of the problem with analyzing CT into skills is that a good deal 
of the knowledge that makes up CT (quite apart from the so-called knowledge 
which it presupposes) is propositional knowledge and not a skill. One cannot 
spot an argument or assess it validly without being able to define what an 
argument is and what valid is. Forcing CT into skills is basically confused. 
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Department of Philosophy May 7, 1787 


Dear Delphi Colleagues, 


Dozens of responses to rounds 5-A, S-B, and 5-C have come in. Again , 
I find myself admiring your perceptive analyses and sound suggestions. i 
And I am not just saying that to make yon fell good. Your responses 
range from the outstanding to the excellent. Often I find myself writing 
"good point" and "remember" in the margins of your letters. 


Some of you have not responded yet the ROUND FIVE letters. But the 
notes ar phone calls from thase too busy to jump in to round five were 
certainly appreciated. If you still have round five on you desk and are 
wondering whether or not to respond now that the semester is coming to an 
end the answer is please, by all means. We need your input not just to 
keep up the quantity of participants but also for your ideas and 
insights. There is still time to reply before I start putting together 
the draft of our final report, a project I will not undertake until 
August or September. For tne summer I've taken on a huge teaching load 
to help three of my children and my wife continue their undergraduate and 
graduate educations. 


I hope to make two presentations at the Sonoma State conference in 
August. One is a workshop on CT assessment, the other is tentatively 
called "The Marlboro Man and Broad-Shouldered CT" The paper, "Assessing 
CT and Building Consensus on CT," delivered at the Pacific Division 
meetings of the AFA went well. I’m doing a similar presentation for the 
Education Colloquium of UC Davis next week and again in Baltimore in 
Sept. A couple of junior colleges and a couple of state universities in 
California have asked me tu do staff development presentations on CT 
assessment. A favorite is "Thirty Ways to Mess Up a CT Test," another 
is "What is an Elephant, or Defining CT for Practical Purposes." 


The above is a not too subtle way of saying I’m available if you are 
Planning something on. CT. 


CSU Fullerton has given me some time next academic year to pilot 
test a CT assessment instrument. The curriculum committee at our 
university defines CT based on CSU executive order 338. For those of you 
outside CA, that executive order gives a general specification of CT for 
Purposes of requiring a "CT course" be included in the general education 
program of all twenty of state universities. That definition is not 
inherently inconsistent with our Delphi findings, but it is far. less 
sophisticated. If my research project goes along as planned, then by 
this time next year I should know if the CT assessment tool we will have 
put together is capable of detecting improvements in students’ CT which 
result from their taking a 3 unit required lower division CT course from 
the Philosophy Dept. This is a dangerous question! Ironically, a key 
assumption in our experimental design is that the Philosophy faculty 
teaching CT are doing an effective job. The experiment is to find out if 
the assessment tool is sensitive enough to detect the difference we 
assume our CT instruction is making. 
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The California State Univarcity pC EL A 2 a 
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Have a good summer. 
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California State University, Fullerton 
Fullerton, California 92634-4080 





Department of Philosophy uk t ee 
(714) 773-3611 FINAL ROUND 


DELPHI ROUND 6 


Sept. 25, 1989 
Dear Delphi Colleagues, 





Please review the enclosed final draft. Your 
your comments, substantive or editorial, are 
vital. Be sure to check the consensus 


statements in the Tables and the final 
recommendations. TO BE HELPFUL YOUR COMMENTS 
AND SUGGESTIONS MUST REACH ME BY OCT. 30. 


The final report to the APA Committee on Pre-College 
Fhilosophy is due in November. Its intended audience is 
educators, at any level, interested in CT. To clarify. as 
principle investigator my role is to express the Delphi findings 
obiectively, whether they be points of consensus, majority 
positions, or minority opinions. I am not one of the 46 expert 
participants. I hope you find the draft a clear and fair-minded 
expression of the delicate confluence of your expert opinions. 


ROUNDS SA, 5B and 5C were extremely useful. Of the 46 
experts, 26, 23 and 22 responded in these rounds. ROUND 5A 
confirmed the strong consensus regarding the cognitive skill 
dimension of CT. ROUND 5B showed that 61% of tbe experts hold 
that CT includes affective dispositions. but 30% maintain CT does 
not. However, over 80% would be willing to use the same list of 
affective dispositions in describing the paradigm critical 
thinker. Although everyone recognized the personal and civic 
value of CT, only 17% argued that "CT" has a normative meaning. 
ROUND 5C produced several thoughtful recommendations relating to 
CT teaching and assessment. Be sure to consider these carerully 
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or college educators. 


In Delphi research once an expert expresses an opinion, even 
a dissenting one, it becomes a factor in the mix and flow of all 
subsequent argument and thought. Because of this, and because in 
Delphi research it is reasonable to assume that silence from busy 
experts is a sign of general accord with the direction of inquiry, 
I will continue to operate on the principle that unless you 
object, you find things generally acceptable. 


My respect for your collective and individual wisdom and my 
gratitude for your participation are so deep that I cannot 
possibly express either as fully as I feel them. Thank you se 
very much for being part of this two year adventure. I hope the 





I am most appreciatively yours. ; 
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